NEST和TPL数据流

Question

问题

我们前后编写了一个日志记录框架，它使用 NEST 将日志索引到 ElasticSearch 。目前，我正在使用NEST提供的BulkAsync(...)调用升级框架以执行批量插入。我们认为使用 TPL数据流来构建批量请求（使用BatchBlock和ActionBlock）然后在批量目标被点击时，将其运送到批量索引方法。问题是当执行BulkAsync或任何其他索引方法时，在ActionBlock内引发异常（使用消息"Thread was being aborted."）。此消息和堆栈跟踪是我可以获得异常的唯一信息，因为visual studio无法在xUnit的调试模式中显示异常，基本上我得到（"Unable to evaluate expression because the code is optimized or a native frame is on top of the call stack"）但是这是无关的

堆栈跟踪

at System.Runtime.CompilerServices.RuntimeHelpers.GetHashCode(Object o)\r\n   
at System.RuntimeType.GetHashCode()\r\n   
at System.Collections.Generic.ObjectEqualityComparer`1.GetHashCode(T obj)\r\n   
at System.Collections.Generic.Dictionary`2.FindEntry(TKey key)\r\n   
at System.Collections.Generic.Dictionary`2.TryGetValue(TKey key, TValue& value)\r\n   
at Newtonsoft.Json.Utilities.ConvertUtils.GetTypeCode(Type t, Boolean& isEnum)\r\n   
at Newtonsoft.Json.Utilities.ConvertUtils.GetTypeCode(Type t)\r\n   
at Newtonsoft.Json.Serialization.DefaultContractResolver.IsJsonPrimitiveType(Type t)\r\n   
at Newtonsoft.Json.Serialization.DefaultContractResolver.CreateContract(Type objectType)\r\n   
at Nest.Resolvers.ElasticContractResolver.CreateContract(Type objectType) in c:\\code\\elasticsearch-net\\src\\Nest\\Resolvers\\ElasticContractResolver.cs:line 30\r\n   
at Newtonsoft.Json.Serialization.DefaultContractResolver.ResolveContract(Type type)\r\n   
at Nest.Resolvers.SettingsContractResolver.ResolveContract(Type type) in c:\\code\\elasticsearch-net\\src\\Nest\\Resolvers\\SettingsContractResolver.cs:line 34\r\n   
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.GetContractSafe(Object value)\r\n   
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)\r\n   
at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)\r\n   
at Newtonsoft.Json.JsonSerializer.Serialize(JsonWriter jsonWriter, Object value, Type objectType)\r\n   
at Newtonsoft.Json.JsonConvert.SerializeObjectInternal(Object value, Type type, JsonSerializer jsonSerializer)\r\n   
at Newtonsoft.Json.JsonConvert.SerializeObject(Object value, Type type, Formatting formatting, JsonSerializerSettings settings)\r\n   
at Newtonsoft.Json.JsonConvert.SerializeObject(Object value, Formatting formatting, JsonSerializerSettings settings)\r\n   
at Nest.NestSerializer.Serialize(Object data, SerializationFormatting formatting) in c:\\code\\elasticsearch-net\\src\\Nest\\ExposedInternals\\NestSerializer.cs:line 30\r\n   
at Nest.NestSerializer.SerializeBulkDescriptor(IBulkRequest bulkRequest) in c:\\code\\elasticsearch-net\\src\\Nest\\ExposedInternals\\NestSerializer.cs:line 175\r\n   
at Nest.ElasticClient.<BulkAsync>b__345_0(ElasticsearchPathInfo`1 p, IBulkRequest d) in c:\\code\\elasticsearch-net\\src\\Nest\\ElasticClient-Bulk.cs:line 44\r\n   
at Nest.ElasticClient.Nest.IHighLevelToLowLevelDispatcher.DispatchAsync[D,Q,R,I](D descriptor, Func`3 dispatch) in c:\\code\\elasticsearch-net\\src\\Nest\\ElasticClient.cs:line 88\r\n   
at Nest.ElasticClient.BulkAsync(IBulkRequest bulkRequest) in c:\\code\\elasticsearch-net\\src\\Nest\\ElasticClient-Bulk.cs:line 40\r\n   
at ElasticSearchLogger.<IndexLogEntriesAsync>d__f.MoveNext() in d:\\MyProject\\ElasticSearchLogger.cs:line 218

我尝试了什么

我已经在ActionBlock之外测试了批量索引代码并且它有效，它只是在块内部失败了。 有关为何会发生这种情况的任何想法？

工作代码

不使用ActionBlock来调用它

private async Task IndexLogEntriesAsync(IEnumerable<LogEntry> logEntries)
{
    try
    {
        var request = new BulkDescriptor();
        foreach (LogEntry logEntry in logEntries)
            request.Index<LogEntry>(op => op
                    .Index(_index)
                    .Type("logentry")
                    .Id(Guid.NewGuid().ToString())
                    .Document(logEntry));

        var result = await _elasticClient.BulkAsync(request);
    }
    catch (Exception e)
    {
        string m = e.Message; // Thread was being aborted.
        // No can do this logging thing.
    }
}

什么打破

BatchBlock<LogEntry> batchBlock = new BatchBlock<LogEntry>(_batchSize); //batch size = 10
ActionBlock<IEnumerable<LogEntry>> asyncIndexer = new ActionBlock<IEnumerable<LogEntry>>(
    async (le) =>
    {
        await this.IndexLogEntriesAsync(le);
    });

batchBlock.LinkTo(asyncIndexer, new DataflowLinkOptions { PropagateCompletion = true });

//And in the IndexLogEntry method call
batchBlock.Post(logEntry); //which builds up the batch and ships when the batch is full

更新

我摆脱了TPL Data流程块并编写了我自己的解决方案，以测试不同的场景。使用ConcurrentQueue不断对项目进行排队并在超时或出现批量大小时从一个线程中出列。我得到了完全相同的错误。