Work with huge list of entities in batches? It has been never easier.
Common programming issue –you have list of 10000 integers or entities (bad programming practice!) in generic list and you need to process them in batches of, let’s say, 250 items per batch.
In the past I solved this problem with following algorithm:
- Sort items per some criteria (in most case sort on identifier)
- Take first chunk of data
- Save last processed id into local variable
- While batch size is more than zero
- Process batch
- Take new chunk of data where identifiers are greater than last processed one
- Set new last processed id
Here is listed code snippet:
//sort list
adIds.Sort((number1, number2) => number1.CompareTo(number2));
int lastProcessedId = 0;
List<> tempList = (adIds.Where(item => item > lastProcessedId)).Take(250).ToList();
while (tempList.Count() > 0)
{
markerAds = AdHandler.GetByIDList(tempList); //items processing (in this case grabbing data from db)
lastProcessedId = tempList[tempList.Count - 1];
tempList = (adIds.Where(item => item < lastProcessedId)).Take(250).ToList();
It looks like few simple lines of code. By power of linq, anonymous types and group by method in c# 3.0 it can be done even simpler and in less lines of code.
var groups = adIds.Select((id, index) => new {GroupID = index/250, ID = id}).GroupBy(x => x.GroupID);
foreach (var group in groups)
{
markerAds = AdHandler.GetByIDList(group.Select(x=>x.ID).ToList());
}
Variable “groups” contains chunks of 250 items per group. Faster, simpler and better.
Cheers
