Question

我有一个使用2个sql数据库的过程 - 这是将一些数据从DB1迁移到DB2所必需的。要迁移的记录数约为500k。要连接到这两个数据库，我使用EF和2个edmx文件。 BackgroundWorker用于执行此操作。

代码非常简单，即：

string userState = "(Current action description)";
int idx = 0;
foreach (var o in DB1.Records)
{
    if (((BackgroundWorker)sender.CancellationPending)
    {
        e.Cancel = true;
        break;
    }

    // Check if the current record already exists in the new database, update it if it does 
    // and create a new one if it doesn't.
    NewRecord rec = DB2.NewRecords.Where(x => x.IDFromDB1 == o.ID).SingleOrDefault() ?? DB2.NewRecords.Where(x => x.UniqueID == o.UniqueID).SingleOrDefault();
    if (rec == null)
        rec = new NewRecord();

    // Since the primary key in Records table of DB1 database consists of 2 columns,
    // where ID is the first and Idx is the second, in a way that Idx marks the revision
    // and we want to select only the latest revision, we order by "Idx" in descending order
    // and take the first record.
    Record obj = DB1.Records.Where(x => x.ID == o.ID).OrderByDescending(x => x.Idx).First(); // <-- System.OutOfMemoryException after around 250k records have been processed

    /*
        ... some primitive processing code ...
    */

    if (rec.EntityKey == null)
        DB2.NewRecords.AddObject(rec);

    idx++;
    if (idx % 50 == 0)
    {
        DB2.SaveChanges();
        ((BackgroundWorker)sender).ReportProgress(idx, userState);
    }
}

DB2.SaveChanges();

该程序在处理了约250,000条记录后引发了例外为什么？有没有更好的方法呢？

堆栈跟踪

at System.String.Concat(String str0, String str1)
at System.Data.Metadata.Edm.EdmType.CreateEdmTypeIdentity(String namespaceName, String name)
at System.Data.Metadata.Edm.EdmType.BuildIdentity(StringBuilder builder)
at System.Data.Metadata.Edm.EdmType.get_Identity()
at System.Data.Metadata.Edm.TypeUsage.BuildIdentity(StringBuilder builder)
at System.Data.Metadata.Edm.CollectionType.GetIdentity(TypeUsage typeUsage)
at System.Data.Metadata.Edm.CollectionType..ctor(TypeUsage elementType)
at System.Data.Common.CommandTrees.ExpressionBuilder.Internal.ArgumentValidation.ValidateProject(DbExpressionBinding input, DbExpression projection)
at System.Data.Query.PlanCompiler.CTreeGenerator.CreateProject(RelOpInfo sourceInfo, IEnumerable`1 outputVars)
at System.Data.Query.PlanCompiler.CTreeGenerator.BuildProjection(Node relOpNode, IEnumerable`1 projectionVars)
at System.Data.Query.PlanCompiler.CTreeGenerator.Visit(PhysicalProjectOp op, Node n)
at System.Data.Query.InternalTrees.PhysicalProjectOp.Accept[TResultType](BasicOpVisitorOfT`1 v, Node n)
at System.Data.Query.InternalTrees.BasicOpVisitorOfT`1.VisitNode(Node n)
at System.Data.Query.PlanCompiler.CTreeGenerator..ctor(Command itree, Node toConvert)
at System.Data.Query.PlanCompiler.ProviderCommandInfoUtils.Create(Command command, Node node, List`1 children)
at System.Data.Query.PlanCompiler.CodeGen.Process(List`1& childCommands, ColumnMap& resultColumnMap, Int32& columnCount)
at System.Data.Query.PlanCompiler.PlanCompiler.Compile(List`1& providerCommands, ColumnMap& resultColumnMap, Int32& columnCount, Set`1& entitySets)
at System.Data.EntityClient.EntityCommandDefinition..ctor(DbProviderFactory storeProviderFactory, DbCommandTree commandTree)
at System.Data.EntityClient.EntityProviderServices.CreateCommandDefinition(DbProviderFactory storeProviderFactory, DbCommandTree commandTree)
at System.Data.EntityClient.EntityProviderServices.CreateDbCommandDefinition(DbProviderManifest providerManifest, DbCommandTree commandTree)
at System.Data.Common.DbProviderServices.CreateCommandDefinition(DbCommandTree com mandTree)
at System.Data.Objects.Internal.ObjectQueryExecutionPlan.Prepare(ObjectContext context, DbQueryCommandTree tree, Type elementType, MergeOption mergeOption, Span span, ReadOnlyCollection`1 compiledQueryParameters)
at System.Data.Objects.ELinq.ELinqQueryState.GetExecutionPlan(Nullable`1 forMergeOption)
at System.Data.Objects.ObjectQuery`1.GetResults(Nullable`1 forMergeOption)
at System.Data.Objects.ObjectQuery`1.System.Collections.Generic.IEnumerable<T>.GetEnumerator()
at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
at System.Data.Objects.ELinq.ObjectQueryProvider.<GetElementFunction>b__0[TResult](IEnumerable`1 sequence)
at System.Data.Objects.ELinq.ObjectQueryProvider.ExecuteSingle[TResult](IEnumerable`1 query, Expression queryRoot)
at System.Data.Objects.ELinq.ObjectQueryProvider.System.Linq.IQueryProvider.Execute[S](Expression expression)
at System.Linq.Queryable.First[TSource](IQueryable`1 source)
at MyProject.frmPrenosIzAnalitike._worker_DoWork(Object sender, DoWorkEventArgs e) in C:\Users\dejan\Documents\Visual Studio 2010\Projects\MySolution\MyProject\frmMyForm.cs:line 245
at System.ComponentModel.BackgroundWorker.OnDoWork(DoWorkEventArgs e)
at System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)

Answer 1

这是因为您一直在向上下文添加新实体。你基本上不想在内存中加载250K实体。

当你不再需要它们时，只需将它们分开

// Assuming DB1 is your object context
Record obj = DB1.Records.Where(x => x.ID == o.ID).OrderByDescending(x => x.Idx).First();

...

DB1.Detach(obj); // <------ Detaches from context, removes from memory

DB2和创建的实体也是如此。

长期运行的foreach循环中的OutOfMemoryException

1 个答案: