Question

我正在使用实体框架并经常遇到问题，我想迭代大量的记录。我的问题是，如果我把它们全部拉出来，我就冒险了;如果我一次拉一个，实际上每个记录都是一个单独的查询，它需要永远。

我想实现一个Linq扩展，它可以批量提取结果，但仍然可以用作IEnumerable。我会给它一组键（很可能是我所拉的任何记录的主要ID），批量大小（对于简单对象更高，对于复杂对象更低），以及定义如何的Func将一组键应用于一组记录类型T。我会这样称呼它：

//get the list of items to pull--in this case, a set of order numbers
List<int> orderNumbers = GetOrderNumbers();

//set the batch size
int batchSize = 100;

//loop through the set using BatchedSelector extension. Note the selection
//function at the end which allows me to 
foreach (var order in dbContext.Orders.BatchedSelector(repairNumbers, batchSize, (o, k) => k.Contains(o.OrderNumber)))
{
    //do things
}

这是我的解决方案草案：

    /// <summary>
    /// A Linq extension that fetches IEnumerable results in batches, aggregating queries
    /// to improve EF performance. Operates transparently to application and acts like any
    /// other IEnumerable.
    /// </summary>
    /// <typeparam name="T">Header record type</typeparam>
    /// <param name="source">Full set of records</param>
    /// <param name="keys">The set of keys that represent specific records to pull</param>
    /// <param name="selector">Function that filters the result set to only those which match the key set</param>
    /// /// <param name="maxBatchSize">Maximum number of records to pull in one query</param>
    /// <returns></returns>
    public static IEnumerable<T> BatchedSelector<T>(this IEnumerable<T> source, IEnumerable<int> keys, Func<T, IEnumerable<int>, bool> selector, int maxBatchSize)
    {
        //the index of the next key (or set of keys) to process--we start at 0 of course
        int currentKeyIndex = 0;             

        //to provide some resiliance, we will allow the batch size to decrease if we encounter errors
        int currentBatchSize = maxBatchSize;
        int batchDecreaseAmount = Math.Max(1, maxBatchSize / 10); //10%, but at least 1

        //other starting variables; a list to hold results and the associated batch of keys
        List<T> resultList = null;
        IEnumerable<int> keyBatch = null;

        //while there are still keys remaining, grab the next set of keys
        while ((keyBatch = keys.Skip(currentKeyIndex).Take(currentBatchSize)).Count() > 0)
        {
            //try to fetch the results
            try
            {
                resultList = source.Where(o => selector(o, keyBatch)).ToList();  // <-- this is where errors occur
                currentKeyIndex += maxBatchSize;  //increment key index to mark these keys as processed
            }
            catch
            {
                //decrease the batch size for our retry
                currentBatchSize -= batchDecreaseAmount;

                //if we've run out of batch overhead, throw the error
                if (currentBatchSize <= 0) throw;

                //otherwise, restart the loop
                continue;
            }

            //since we've successfully gotten the set of keys, yield the results
            foreach (var match in resultList) yield return match;
        }

        //the loop is over; we're done
        yield break;
    }

出于某种原因，＆＃34;其中＆＃34;条款没有效果。我已经验证了正确的密钥位于keyBatch中，但预期的WHERE OrderNumber IN (k1, k2, k3, kn)行不存在。好像我根本没有where声明。

我最好的猜测是我需要构建表达式并对其进行编译，但我不确定这是否是问题所在并且我不确定如何修复它。会喜欢任何输入。谢谢！

Answer 1

Where，Skip，Take所有这些方法都是扩展方法，而不是IEnumerable<T>的成员。对于所有这些方法，实际上有两个版本，一个用于IEnumerable<>，另一个用于IQueryable<>。

可枚举的扩展程序

Where(Func<TSource, bool> predicate)
Select(Func<TSource, TResult> selector)

可查询扩展程序

Where(Expression<Func<TSource, bool>> predicate)
Select(Expression<Func<TSource, TResult>> predicate)

正如您所看到的那样，区别在于Queryable个扩展名采用Expression<>而不是直接委托。这些表达式允许 EF 将代码转换为 SQL 。

由于您在BatchedSelector()方法中将变量/参数声明为IEnumerable<>，因此您使用Enumerable类中的扩展名，并且此扩展名在内存中执行。

一个常见的错误是认为由于多态性而导致DbSet（IQueryable<>），无论您将其用作IEnumerable<>，查询都将被转换为 SQL ，这仅适用于正确的成员，但不适用于扩展方法。

您的代码可以修改，将您的IEnumerable<>变量/参数更改为IQueryable<>。

您可以详细了解IEnumerable和IQueryable here之间的差异。

IEnumerable扩展可以批量生成结果

1 个答案: