Question

我们有一个应用程序在晚上进行一些处理。简而言之，它为每个用户创建了一些统计信息（大约10.000）。

现在我们注意到这在生产环境中需要数小时，我们已经能够使用生产数据库的备份来模拟这一点。

我们看到，当foreach循环开始时，生成数据通常需要大约200 ms，并将其保存到数据库中以供一个用户。在大约1000个用户之后，每个用户最多可达700毫秒。在大约2.000个用户之后，它开始花费的时间越来越长，一直长达2秒，以生成数据并将其保存到每个用户的数据库中。

你有什么想法可能会这样吗？以下是显示发生情况的（简化）代码：

var userService = IoC.GetInstance<IUserService>();
var users  = userService.GetAll().Where(x => x.IsRegistered == true);

var statisticsService = IoC.GetInstance<IStatisticsService>();

foreach (var user in users.ToList())
{
   var statistic1 = GetStatistic(1); // this returns an object
   var statistic 2 = GetStatistic(2);

   statisticsService.Add(statistic1);
   statisticsService.Add(statistic2);

   statisticsService.Commit(); /* this is essentially a dbContext.SaveChanges(); */
}

function Statistic GetStatistic(int statisticnumber)
{
   var stat = new Statistic();
   stat.Type = statisticnumber;

   switch(statisticnumber) {
       case 1:
          stat.Value = /* query to count how many times they've logged in */
          break;
       case 2:
          stat.Value = /* query to get their average score */
          break;
       ...
    }
    return stat;
}

到目前为止，我们已经尝试过：

AsNoTracking ：我们选择使用AsNoTracking的用户，以确保Entity Framework不会跟踪用户自身的任何更改（因为没有）。
清除统计信息表上的索引：在此开始之前，我们运行一个删除所有索引的脚本（聚簇索引除外）。生成后，我们重新创建这些索引

有没有人有我们可以测试/尝试的其他东西？

Answer 1

As you can see in comments you need to keep the context clean so you need to Dispose it every n records (usually, in my case, n < 1000).
This is a good solution in most cases. But there are some issues, the most important are:
1. When you need to insert a lot of records, running insert (and update) statements runs faster.
2. The entities you write (and related entities) must be all in the same context.

There are some other libraries around to make bulk operations but they works only with SQL Server and I think that a great added value of EF is that is DBMS independent without significant efforts.

When I need to insert several records (less than 1.000.000) and I want to keep EF advantages I use the following methods. They generates a DML statement starting from an entity.

public int ExecuteInsertCommand(object entityObject)
{
    DbCommand command = GenerateInsertCommand(entityObject);
    ConnectionState oldConnectionState = command.Connection.State;
    try
    {
        if (oldConnectionState != ConnectionState.Open)
            command.Connection.Open();
        int result = command.ExecuteNonQuery();
        return result;

    }
    finally
    {
        if (oldConnectionState != ConnectionState.Open)
            command.Connection.Close();
    }
}

public DbCommand GenerateInsertCommand(object entityObject)
{

    ObjectContext objectContext = ((IObjectContextAdapter)Context).ObjectContext;
    var metadataWorkspace = ((EntityConnection)objectContext.Connection).GetMetadataWorkspace();

    IEnumerable<EntitySetMapping> entitySetMappingCollection = metadataWorkspace.GetItems<EntityContainerMapping>(DataSpace.CSSpace).Single().EntitySetMappings;
    IEnumerable<AssociationSetMapping> associationSetMappingCollection = metadataWorkspace.GetItems<EntityContainerMapping>(DataSpace.CSSpace).Single().AssociationSetMappings;

    var entitySetMappings = entitySetMappingCollection.First(o => o.EntityTypeMappings.Select(e => e.EntityType.Name).Contains(entityObject.GetType().Name));

    var entityTypeMapping = entitySetMappings.EntityTypeMappings[0];
    string tableName = entityTypeMapping.EntitySetMapping.EntitySet.Name;

    MappingFragment mappingFragment = entityTypeMapping.Fragments[0];

    string sqlColumns = string.Empty;
    string sqlValues = string.Empty;
    int paramCount = 0;

    DbCommand command = Context.Database.Connection.CreateCommand();

    foreach (PropertyMapping propertyMapping in mappingFragment.PropertyMappings)
    {
        if (((ScalarPropertyMapping)propertyMapping).Column.StoreGeneratedPattern != StoreGeneratedPattern.None)
            continue;

        string columnName = ((ScalarPropertyMapping)propertyMapping).Column.Name;
        object columnValue = entityObject.GetType().GetProperty(propertyMapping.Property.Name).GetValue(entityObject, null);
        string paramName = string.Format("@p{0}", paramCount);

        if (paramCount != 0)
        {
            sqlColumns += ",";
            sqlValues += ",";
        }

        sqlColumns += SqlQuote(columnName);
        sqlValues += paramName;

        DbParameter parameter = command.CreateParameter();
        parameter.Value = columnValue;
        parameter.ParameterName = paramName;
        command.Parameters.Add(parameter);

        paramCount++;
    }

    foreach (var navigationProperty in entityTypeMapping.EntityType.NavigationProperties)
    {
        PropertyInfo propertyInfo = entityObject.GetType().GetProperty(navigationProperty.Name);
        if (typeof(System.Collections.IEnumerable).IsAssignableFrom(propertyInfo.PropertyType))
            continue;

        AssociationSetMapping associationSetMapping = associationSetMappingCollection.First(a => a.AssociationSet.ElementType.FullName == navigationProperty.RelationshipType.FullName);

        EndPropertyMapping propertyMappings = associationSetMapping.AssociationTypeMapping.MappingFragment.PropertyMappings.Cast<EndPropertyMapping>().First(p => p.AssociationEnd.Name.EndsWith("_Target"));

        object relatedObject = propertyInfo.GetValue(entityObject, null);

        foreach (ScalarPropertyMapping propertyMapping in propertyMappings.PropertyMappings)
        {
            string columnName = propertyMapping.Column.Name;
            string paramName = string.Format("@p{0}", paramCount);
            object columnValue = relatedObject == null ?
                null :
                relatedObject.GetType().GetProperty(propertyMapping.Property.Name).GetValue(relatedObject, null);

            if (paramCount != 0)
            {
                sqlColumns += ",";
                sqlValues += ",";
            }

            sqlColumns += SqlQuote(columnName);
            sqlValues += string.Format("@p{0}", paramCount);

            DbParameter parameter = command.CreateParameter();
            parameter.Value = columnValue;
            parameter.ParameterName = paramName;
            command.Parameters.Add(parameter);

            paramCount++;
        }
    }

    string sql = string.Format("INSERT INTO {0} ({1}) VALUES ({2})", tableName, sqlColumns, sqlValues);
    command.CommandText = sql;

    foreach (DbParameter parameter in command.Parameters)
    {
        if (parameter.Value == null)
            parameter.Value = DBNull.Value;
    }

    return command;
}

public int ExecuteUpdateCommand(object entityObject)
{
    DbCommand command = GenerateUpdateCommand(entityObject);
    ConnectionState oldConnectionState = command.Connection.State;
    try
    {
        if (oldConnectionState != ConnectionState.Open)
            command.Connection.Open();
        int result = command.ExecuteNonQuery();
        return result;
    }
    finally
    {
        if (oldConnectionState != ConnectionState.Open)
            command.Connection.Close();
    }
}

public DbCommand GenerateUpdateCommand(object entityObject)
{

    ObjectContext objectContext = ((IObjectContextAdapter)Context).ObjectContext;
    var metadataWorkspace = ((EntityConnection)objectContext.Connection).GetMetadataWorkspace();

    IEnumerable<EntitySetMapping> entitySetMappingCollection = metadataWorkspace.GetItems<EntityContainerMapping>(DataSpace.CSSpace).Single().EntitySetMappings;
    IEnumerable<AssociationSetMapping> associationSetMappingCollection = metadataWorkspace.GetItems<EntityContainerMapping>(DataSpace.CSSpace).Single().AssociationSetMappings;

    string entityTypeName;
    if (!entityObject.GetType().Namespace.Contains("DynamicProxi"))
        entityTypeName = entityObject.GetType().Name;
    else
        entityTypeName = entityObject.GetType().BaseType.Name;
    var entitySetMappings = entitySetMappingCollection.First(o => o.EntityTypeMappings.Select(e => e.EntityType.Name).Contains(entityTypeName));

    var entityTypeMapping = entitySetMappings.EntityTypeMappings[0];
    string tableName = entityTypeMapping.EntitySetMapping.EntitySet.Name;

    MappingFragment mappingFragment = entityTypeMapping.Fragments[0];

    string sqlColumns = string.Empty;
    int paramCount = 0;

    DbCommand command = Context.Database.Connection.CreateCommand();

    foreach (PropertyMapping propertyMapping in mappingFragment.PropertyMappings)
    {
        if (((ScalarPropertyMapping)propertyMapping).Column.StoreGeneratedPattern != StoreGeneratedPattern.None)
            continue;

        string columnName = ((ScalarPropertyMapping)propertyMapping).Column.Name;

        if (entityTypeMapping.EntityType.KeyProperties.Select(_ => _.Name).Contains(columnName))
            continue;

        object columnValue = entityObject.GetType().GetProperty(propertyMapping.Property.Name).GetValue(entityObject, null);
        string paramName = string.Format("@p{0}", paramCount);

        if (paramCount != 0)
            sqlColumns += ",";

        sqlColumns += string.Format("{0} = {1}", SqlQuote(columnName), paramName);

        DbParameter parameter = command.CreateParameter();
        parameter.Value = columnValue ?? DBNull.Value;
        parameter.ParameterName = paramName;
        command.Parameters.Add(parameter);

        paramCount++;
    }

    foreach (var navigationProperty in entityTypeMapping.EntityType.NavigationProperties)
    {
        PropertyInfo propertyInfo = entityObject.GetType().GetProperty(navigationProperty.Name);
        if (typeof(System.Collections.IEnumerable).IsAssignableFrom(propertyInfo.PropertyType))
            continue;

        AssociationSetMapping associationSetMapping = associationSetMappingCollection.First(a => a.AssociationSet.ElementType.FullName == navigationProperty.RelationshipType.FullName);

        EndPropertyMapping propertyMappings = associationSetMapping.AssociationTypeMapping.MappingFragment.PropertyMappings.Cast<EndPropertyMapping>().First(p => p.AssociationEnd.Name.EndsWith("_Target"));

        object relatedObject = propertyInfo.GetValue(entityObject, null);


        foreach (ScalarPropertyMapping propertyMapping in propertyMappings.PropertyMappings)
        {
            string columnName = propertyMapping.Column.Name;
            string paramName = string.Format("@p{0}", paramCount);
            object columnValue = relatedObject == null ?
                null :
                relatedObject.GetType().GetProperty(propertyMapping.Property.Name).GetValue(relatedObject, null);

            if (paramCount != 0)
                sqlColumns += ",";

            sqlColumns += string.Format("{0} = {1}", SqlQuote(columnName), paramName);

            DbParameter parameter = command.CreateParameter();
            parameter.Value = columnValue ?? DBNull.Value;
            parameter.ParameterName = paramName;
            command.Parameters.Add(parameter);

            paramCount++;
        }
    }


    string sqlWhere = string.Empty;
    bool first = true;
    foreach (EdmProperty keyProperty in entityTypeMapping.EntityType.KeyProperties)
    {
        var propertyMapping = mappingFragment.PropertyMappings.First(p => p.Property.Name == keyProperty.Name);
        string columnName = ((ScalarPropertyMapping)propertyMapping).Column.Name;
        object columnValue = entityObject.GetType().GetProperty(propertyMapping.Property.Name).GetValue(entityObject, null);
        string paramName = string.Format("@p{0}", paramCount);

        if (first)
            first = false;
        else
            sqlWhere += " AND ";

        sqlWhere += string.Format("{0} = {1}", SqlQuote(columnName), paramName);

        DbParameter parameter = command.CreateParameter();
        parameter.Value = columnValue;
        parameter.ParameterName = paramName;
        command.Parameters.Add(parameter);

        paramCount++;

    }


    string sql = string.Format("UPDATE {0} SET {1} WHERE {2}", tableName, sqlColumns, sqlWhere);
    command.CommandText = sql;

    return command;
}

Answer 2

添加方法

每次迭代后，此方法变得越来越慢。实际上，这种方法并不慢，但是在Add方法中调用了DetectChanges方法。

ChangeTracker包含的记录越多，DetectChanges方法变得越慢。

在大约100,000个实体中，添加第一个实体时，只需添加一个新实体就可以获得超过200毫秒。

<强>解决方案

有几种解决方案可以解决此问题，例如：

使用AddRange over Add
SET AutoDetectChanges to false
SPLIT多个批次的SaveChanges

在你的情况下，每次循环时重新创建一个新的上下文可能是最好的主意，因为它看起来你想要在每次迭代时保存。

foreach (var user in users.ToList())
{
   var statisticsService = new Instance<IStatisticsService>();

   var statistic1 = GetStatistic(1); // this returns an object
   var statistic 2 = GetStatistic(2);

   statisticsService.Add(statistic1);
   statisticsService.Add(statistic2);

   statisticsService.Commit(); /* this is essentially a dbContext.SaveChanges(); */
}

一旦你摆脱了由于DetectChanges方法导致的糟糕性能，你仍然会遇到由SaveChanges方法执行的数据库往返次数引起的性能问题。

如果您需要保存10,000个统计信息，那么SaveChanges将进行10,000次数据库往返，这是 INSANELY 慢。

免责声明：我是该项目的所有者Entity Framework Extensions

此库允许您执行所有批量操作：

BulkSaveChanges
BulkInsert
BulkUpdate
BulkDelete
BulkMerge
BulkSynchronize

它将适用于所有主要提供商：

SQL Server
SQL Compact
甲骨文
MySQL的
SQLite的
的PostgreSQL

示例：

// Easy to use
context.BulkSaveChanges();

// Easy to customize
context.BulkSaveChanges(bulk => bulk.BatchSize = 100);

// Perform Bulk Operations
context.BulkDelete(customers);
context.BulkInsert(customers);
context.BulkUpdate(customers);

// Customize Primary Key
context.BulkMerge(customers, operation => {
   operation.ColumnPrimaryKeyExpression = 
        customer => customer.Code;
});

每次迭代都会使Foreach变慢

2 个答案: