Question

我们发现在我们的服务中对ReliableDictionary.AddOrUpdateAsync的调用偶尔会抛出FabricObjectClosedExceptions。我们的api测试持续运行在这段代码上，但只是失败了一小部分时间，所以它似乎不是我们自己的代码的明显问题，而是底层商店的问题。

被调用的代码是：

 using (var tx = StateManager.CreateTransaction())
 {
      // store the token
      await _tokens.AddOrUpdateAsync(
            tx,
            key,
            value,
            (s, entities) => value);
      await tx.CommitAsync();
 }

如上所述，此代码大部分时间都有效，但在occation上出现FabricObjectClosedException失败。我们查看了日志，但无法将其与我们发现的任何内容相关联。

有人可以确定为什么可以关闭ReliableDictionary以及我们是否需要一些重试逻辑？

AddOrUpdateAsync调用抛出异常，具有以下stacktrace（略有格式化）：

    Exception of type 'System.Fabric.FabricObjectClosedException' was thrown.
 at async Microsoft.ServiceFabric.Services.Communication.Client.ServicePartitionClient`1.InvokeWithRetryAsync[](?)
 at async Microsoft.ServiceFabric.Services.Remoting.Client.ServiceRemotingPartitionClient.InvokeAsync(?)
 at async Microsoft.ServiceFabric.Services.Remoting.Builder.ProxyBase.InvokeAsync(?)
 at async Microsoft.ServiceFabric.Services.Remoting.Builder.ProxyBase.ContinueWith(?)
 at async DataNamespace.Stores.BaseStore`1.StoreAsync[](?) 
 at async IdentityServer3.Core.Services.Default.DefaultTokenService.CreateSecurityTokenAsync(?)
 at async IdentityServer3.Core.ResponseHandling.TokenResponseGenerator.CreateAccessTokenAsync(?)
 at async IdentityServer3.Core.ResponseHandling.TokenResponseGenerator.ProcessTokenRequestAsync(?) 
 at async IdentityServer3.Core.ResponseHandling.TokenResponseGenerator.ProcessAsync(?) 
 at async IdentityServer3.Core.Endpoints.TokenEndpointController.ProcessAsync(?)
 at async IdentityServer3.Core.Endpoints.TokenEndpointController.Post(?) 
 at async System.Threading.Tasks.System.Web.Http908956.TaskHelpersExtensions.CastToObject[](?)
 at async System.Web.Http.Controllers.ApiControllerActionInvoker.InvokeActionAsyncCore(?)
 at async System.Web.Http.Filters.ActionFilterAttribute.CallOnActionExecutedAsync(?)
 at async System.Web.Http.Filters.ActionFilterAttribute.CallOnActionExecutedAsync(?)
 at async System.Web.Http.Filters.ActionFilterAttribute.ExecuteActionFilterAsyncCore(?)
 at async System.Web.Http.Controllers.ActionFilterResult.ExecuteAsync(?)
 at async System.Web.Http.Filters.AuthorizationFilterAttribute.ExecuteAuthorizationFilterAsyncCore(?)
 at async System.Web.Http.Dispatcher.HttpControllerDispatcher.SendAsync(?)
---> (Inner Exception #0) System.Fabric.FabricObjectClosedException: Exception of type 'System.Fabric.FabricObjectClosedException' was thrown.
 at System.Fabric.Store.TStore`5.ThrowIfFaulted(StoreReadWriteTransaction`4 rwtx)
 at System.Fabric.Store.TStore`5.<AddOrUpdateAsync>d__8a.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Microsoft.ServiceFabric.Data.Collections.DistributedDictionary`2.<AddOrUpdateAsync>d__1d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at DataNamespace.TokenPersistenceService.<StoreTokenAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Microsoft.ServiceFabric.Services.Remoting.Builder.MethodDispatcherBase.<ContinueWith>b__3(Task t)
 at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke()
 at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Microsoft.ServiceFabric.Services.Remoting.Builder.MethodDispatcherBase.<Microsoft.ServiceFabric.Services.Remoting.IMethodDispatcher.DispatchAsync>b__5(Task`1 t)
 at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
 at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Microsoft.ServiceFabric.Services.Remoting.Runtime.ServiceRemotingDispatcher.<>c__DisplayClass5.<OnDispatch>b__4(Task`1 t)
 at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
 at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Microsoft.ServiceFabric.Services.Remoting.Runtime.ServiceRemotingCancellationHelper.<DispatchRequest>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Microsoft.ServiceFabric.Services.Communication.FabricTransport.Runtime.NativeServiceCommunicationMessageHandlerBroker.<RequestResponseAsync>d__3.MoveNext()

Answer 1

尝试更改本地群集以使用5个节点而不是1个。

我刚刚使用v2.3.301遇到了同样的问题。无论出于何种原因，我在写入（AddOrUpdateAsync）到IReliableDictionary时至少有50％的时间能够重现问题。

经过一些研究（不多见），我发现一些报告说这种情况只发生在本地集群上。在预感中，我将“模式”从1个改为5个，问题似乎已经消失了。

我没有做太多测试但是无法使用5个群集重现问题。

请注意，一旦我看到FabricObjectClosedException，任何重试次数都不会消失。

从ReliableDictionary

1 个答案: