I have a table Users
with id, this table has a million records or more. So when fetching data from the table I am using batching with offset & limit to reduce load on my server.
My question is if we are executing the below query of fetching the data from the table, simultaneously a query comes up inserting a record in the table. Will there be any consistency issues in data returned. The query will be executed until all the data rows in the table is finished.
Query:
SELECT * FROM Users ORDER BY id LIMIT 10000 OFFSET x;
The query will be run such that all the entries in table is returned.
Sample Code:
async function fetchDataInBatches(model, whereClause, batchSize = 1000) {
let offset = 0;
let moreDataAvailable = true;
let allData = [];
while (moreDataAvailable) {
const results = await model.findAll({
where: whereClause,
limit: batchSize,
offset: offset,
order: [['id', 'ASC']],
});
if (results.length === 0) {
moreDataAvailable = false;
break;
}
allData = allData.concat(results);
offset += batchSize;
}
return allData;
}
You need to sign in to view this answers
Leave feedback about this