Question

我有以下代码很快从数据库返回22,000,000条记录：

var records = from row in dataContext.LogicalMapTable
                          select
                              new
                              {
                                  row.FwId,
                                  row.LpDefId,
                                  row.FlDefMapID
                              };

上面数据库调用之后的代码需要60秒才能运行：

var cache = new Dictionary<string, int>();
foreach (var record in records)
{
   var tempHashCode = record.FirmwareVersionID + "." + record.LogicalParameterDefinitionID;

   cache.Add(tempHashCode, record.FirmwareLogicalDefinitionMapID);
}

return cache;

有没有更好的方法来提高性能？

Answer 1

代码的第二部分并不慢。它只是导致LINQ查询评估，您可以通过先前使用您的查询来看到这一点，例如

var records = (from row in dataContext.LogicalMapTable
                          select
                              new
                              {
                                  row.FwId,
                                  row.LpDefId,
                                  row.FlDefMapID
                              }).ToList();

因此，您的LINQ查询速度很慢，以下是解决问题的方法。

您可能不需要在内存中缓存22M记录。你可以尝试的事情：

分页（采取，跳过）
更改查询以包含特定ID或其他列。例如。在select * ...

select * ... where id in (1,2,3) ...

在数据库中进行大部分分析工作，速度快，不会占用您的应用内存
首选快速带来小数据批量的查询。您可以同时运行其中几个来更新UI的不同位

Answer 2

正如其他人在评论中提到的那样，阅读整个列表是非常低效的。

根据您发布的代码，我假设在将列表加载到“缓存”后，使用Keyware of FirmwareVersionID +“。”查找FirmwareLogicalDefinitionMapID。 + LogicalParameterDefinitionID;

我提高整体性能和内存使用率的建议是实现一个实际的缓存模式，如下所示：

public static class CacheHelper
{
    public static readonly object _SyncLock = new object();
    public static readonly _MemoryCache = MemoryCache.Default;

    public static int GetFirmwareLogicalDefinitionMapID(int firmwareVersionID, int logicalParameterDefinitionID)
    {
        int result = -1;

        // Build up the cache key
        string cacheKey = string.Format("{0}.{1}", firmwareVersionID, logicalParameterDefinitionID);

        // Check if the object is in the cache already
        if(_MemoryCache.Countains(cacheKey))
        {
            // It is, so read it and type cast it
            object cacheObject = _MemoryCache[cacheKey];

            if(cacheObject is int)
            {
                result = (int)cacheObject;
            }
        }
        else
        {
            // The object is not in cache, aquire a sync lock for thread safety         
            lock(_SyncLock)
            {
                // Double check that the object hasnt been put into the cache by another thread.
                if(!_MemoryCache.Countains(cacheKey))
                {
                    // Still not there, now Query the database
                    result = (from i in dataContext.LogicalMapTable
                              where i.FwId == firmwareVersionID && i.LpDefId == logicalParameterDefinitionID
                              select i.FlDefMapID).FirstOrDefault();

                     // Add the results to the cache so that the next operation that asks for this object can read it from ram
                    _MemoryCache.Add(new CacheItem(cacheKey, result), new CacheItemPolicy() { SlidingExpiration = new TimeSpan(0, 5, 0) });
                }
                else
                {
                    // we lost a concurrency race to read the object from source, its in the cache now so read it from there.
                    object cacheObject = _MemoryCache[cacheKey];
                    if(cacheObject is int)
                    {
                        result = (int)cacheObject;
                    }
                }
            }
        }

        // return the results
        return result;
    }
}

您还应该阅读.Net MemoryCache：http://www.codeproject.com/Articles/290935/Using-MemoryCache-in-Net-4-0

希望这有帮助！

如何阅读大型IEnumerable＆lt;＆gt;快速

2 个答案: