我正在EC2实例AWS上运行thrift服务,该实例使用C#编写并在Mono 3.2下运行(使用最新的AWS .net SDK 1.5.28.2)。该服务击中了dynamodb,我在与发电机交谈时不断遇到低级网络错误。我目前的解决方法是只捕获任何AmazonServiceException并再试一次,这似乎工作正常
我的猜测是这可能是某种Mono错误,但我不确定如何调试此问题以找到原因,是否有人可以建议原因或事情进行研究以尝试修复这些异常?
以下是一个示例例外:
2013-08-08 09:54:21.3991 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure ---> System.Net.WebException: Error getting response stream (ReadDone1): ReceiveFailure ---> System.IO.IOException: EndRead failure ---> System.Net.Sockets.SocketException: Connection reset by peer
at System.Net.Sockets.Socket.EndReceive (IAsyncResult result) [0x00000] in <filename unknown>:0
at System.Net.Sockets.NetworkStream.EndRead (IAsyncResult ar) [0x00000] in <filename unknown>:0
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.EndRead (IAsyncResult ar) [0x00000] in <filename unknown>:0
at Mono.Security.Protocol.Tls.SslStreamBase.InternalReadCallback (IAsyncResult result) [0x00000] in <filename unknown>:0
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) [0x00000] in <filename unknown>:0
at System.Net.HttpWebRequest.GetResponse () [0x00000] in <filename unknown>:0
at Amazon.Runtime.AmazonWebServiceClient.getResponseCallback (IAsyncResult result) [0x00000] in <filename unknown>:0
--- End of inner exception stack trace ---
at Amazon.Runtime.AmazonWebServiceClient.handleHttpWebErrorResponse (Amazon.Runtime.Internal.AsyncResult asyncResult, System.Net.WebException we) [0x00000] in <filename unknown>:0
at Amazon.Runtime.AmazonWebServiceClient.getResponseCallback (IAsyncResult result) [0x00000] in <filename unknown>:0
并显示我经常收到此错误:
2013-08-07 20:31:04.4475 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 20:34:14.0017 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 20:45:06.9636 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 20:53:21.4654 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 20:56:16.8788 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 21:14:20.7060 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 21:21:17.3771 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 21:39:09.1383 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 21:57:11.6650 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 22:07:08.2615 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 22:44:16.6803 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 22:53:08.8771 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 22:53:09.1383 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-08 09:37:28.9755 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-08 09:42:28.3976 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-08 09:54:21.3991 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
这是尝试提交的代码,ddb.BatchWriteItem调用引发错误
private void BatchPutWithRetry(AmazonDynamoDBClient ddb, string table, IEnumerable<Dictionary<string, AttributeValue>> items)
{
var request = new BatchWriteItemRequest
{
RequestItems = new Dictionary<string, List<WriteRequest>>{
{table, items.Select(item => new WriteRequest{ PutRequest = new PutRequest{ Item = item }}).ToList()},
},
};
Log.Trace("BatchWriteItem {0}", request.RequestItems.First().Value.Count);
var totalSends = 0;
var unsentItems = request.RequestItems;
while (true)
{
try
{
totalSends++;
request.RequestItems = unsentItems;
var response = ddb.BatchWriteItem(request);
unsentItems = response.BatchWriteItemResult.UnprocessedItems;
}
catch (ProvisionedThroughputExceededException)
{
Log.Warn("ProvisionThroughputExceeded");
}
catch (Amazon.Runtime.AmazonServiceException ex)
{
if (totalSends > 1000)
throw;
Log.WarnException("Got service error", ex);
}
if (unsentItems.Count == 0)
return;
var delay = TimeSpan.FromMilliseconds(Random.Next(500, 3000));
Log.Trace("Unprocessed {0}, waiting {1:0.000} seconds",
unsentItems.First().Value.Count,
delay.TotalSeconds);
Thread.Sleep(delay);
}
}