Azure Redis缓存 - 多个错误TimeoutException：超时执行GET {key}

时间：2015-04-10 20:18:03

标签： caching azure redis stackexchange.redis azure-redis-cache

我们将应用程序部署到Azure。它使用Azure Redis缓存，我们遇到了很多超时。即：

[TimeoutException: Timeout performing GET textobjectDetails__23290_TextObject, inst: 1, mgr: Inactive, queue: 5, qu=0, qs=5, qc=0, wr=0/0, in=56864/0]

[TimeoutException: Timeout performing GET featured_series_CachedSeries, inst: 1, mgr: Inactive, queue: 4, qu=0, qs=4, qc=0, wr=0/0, in=44470/0]

[TimeoutException: Timeout performing GET SeriesByFranchiseId_1_CachedSeries, inst: 1, mgr: Inactive, queue: 3, qu=0, qs=3, qc=0, wr=0/0, in=11252/0]

[TimeoutException: Timeout performing GET media_silo-1-1-0_Media, inst: 1, mgr: Inactive, queue: 3, qu=0, qs=3, qc=0, wr=0/0, in=15188/0]

[TimeoutException: Timeout performing GET textobjectDetails__3092_TextObject, inst: 3, mgr: Inactive, queue: 7, qu=0, qs=7, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET textobjectbytype_104__TextObject, inst: 11, mgr: Inactive, queue: 9, qu=0, qs=9, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET groupnews_2_14_TextObject, inst: 1, mgr: Inactive, queue: 7, qu=0, qs=7, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET archived_news_by_group_13586_1_TextObject, inst: 2, mgr: Inactive, queue: 7, qu=0, qs=7, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET textobjectDetails__24404_TextObject, inst: 11, mgr: Inactive, queue: 12, qu=0, qs=12, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET standings_232_lcds_fixtures, inst: 2, mgr: Inactive, queue: 11, qu=0, qs=11, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET player_name1099_Player, inst: 4, mgr: Inactive, queue: 11, qu=0, qs=11, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET groupnews_1_6_TextObject, inst: 4, mgr: Inactive, queue: 9, qu=0, qs=9, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET archivednewscount__20789_TextObject, inst: 2, mgr: Inactive, queue: 11, qu=0, qs=11, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET media_id3648_Media, inst: 1, mgr: Inactive, queue: 10, qu=0, qs=10, qc=0, wr=0/0, in=65536/0]

异常的主体对所有人都是一样的：

StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server):509
    StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server):25
    StackExchange.Redis.RedisDatabase.StringGet(RedisKey key, CommandFlags flags):16
    AB.SiteCaching.Providers.RedisDataSource+<>c__DisplayClasse`1.<RetrieveCacheObject>b__b():0
    Microsoft.Practices.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func):115
    AB.SiteCaching.Providers.RedisDataSource.RetrieveCacheObject[T](String fullCacheKey):56
    AB.SiteCaching.Providers.RedisDataSource.RetrieveCached[T](String key, Func`1 onNotCached, TimeSpan timeOut):61
    DataAccess.Data.Caching.CachedSeries.GetSeriesByFranchiseId(Int32 franchiseId):64
    Shared.Services.SeriesService.LatestYearByFranchiseId(Int32 franchiseId):0
    AllBlacksdotcom.Controllers.FixturesController._MostRecentYearOfFixtures(Int32 franchiseId):0
    (unknown).lambda_method(Closure , ControllerBase , Object[] ):-1
    System.Web.Mvc.ActionMethodDispatcher.Execute(ControllerBase controller, Object[] parameters):0
    System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters):87
    System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters):0
    System.Web.Mvc.Async.AsyncControllerActionInvoker.<BeginInvokeSynchronousActionMethod>b__39(IAsyncResult asyncResult, ActionInvocation innerInvokeState):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResult`2.CallEndDelegate(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
    System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncControllerActionInvoker+AsyncInvocationWithFilters.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3d():20
    System.Web.Mvc.Async.AsyncControllerActionInvoker+AsyncInvocationWithFilters+<>c__DisplayClass46.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3f():134
    System.Web.Mvc.Async.AsyncControllerActionInvoker+<>c__DisplayClass33.<BeginInvokeActionMethodWithFilters>b__32(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResult`1.CallEndDelegate(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
    System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethodWithFilters(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncControllerActionInvoker+<>c__DisplayClass21+<>c__DisplayClass2b.<BeginInvokeAction>b__1c():0
    System.Web.Mvc.Async.AsyncControllerActionInvoker+<>c__DisplayClass21.<BeginInvokeAction>b__1e(IAsyncResult asyncResult):65
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResult`1.CallEndDelegate(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
    System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeAction(IAsyncResult asyncResult):0
    System.Web.Mvc.Controller.<BeginExecuteCore>b__1d(IAsyncResult asyncResult, ExecuteCoreState innerState):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
    System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult):0
    System.Web.Mvc.Controller.<BeginExecute>b__15(IAsyncResult asyncResult, Controller controller):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
    System.Web.Mvc.Controller.EndExecute(IAsyncResult asyncResult):0
    System.Web.Mvc.Controller.System.Web.Mvc.Async.IAsyncController.EndExecute(IAsyncResult asyncResult):0
    System.Web.Mvc.MvcHandler.<BeginProcessRequest>b__5(IAsyncResult asyncResult, ProcessRequestState innerState):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
    System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult):0
    System.Web.Mvc.MvcHandler.System.Web.IHttpAsyncHandler.EndProcessRequest(IAsyncResult result):0
    System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerAsyncWrapper+<>c__DisplayClassa.<EndProcessRequest>b__9():0
    System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerWrapper+<>c__DisplayClass4.<Wrap>b__3():0
    System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerWrapper.Wrap[TResult](Func`1 func):0
    System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerWrapper.Wrap(Action action):25
    System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerAsyncWrapper.EndProcessRequest(IAsyncResult result):32
    System.Web.HttpServerUtility.ExecuteInternal(IHttpHandler handler, TextWriter writer, Boolean preserveForm, Boolean setPreviousPage, VirtualPath path, VirtualPath filePath, String physPath, Exception error, String queryStringOverride):772

请参阅我们的超时设置：

retryTimeoutInMilliseconds = "5000" 
connectionTimeoutInMilliseconds = "5000" 
operationTimeoutInMilliseconds = "1000"

我如何处理这些超时？增加operationTimeoutInMilliseconds可以做到吗？我还读到有关g-zip压缩有助于减少从Redis读取数据所需的时间。

所有与Redis相关的nuget软件包都在其最新版本。我们正在使用Azure版Redis Cache的C1版本（会增加到C2帮助？）。

3 个答案:

答案 0 :(得分：5)

改善我们情况的几点：

Protobuf-net而不是BinaryFormatter

我建议使用protobuf-net，因为它会减少要存储在缓存中的值的大小。

<input type="radio" value="1" id="PCELL" class="item" />
<label for="PCELL">PCELL Data</label>

实施重试策略

实现此RedisCacheTransientErrorDetectionStrategy以处理超时问题。

public interface ICacheDataSerializer
    {
        byte[] Serialize(object o);
        T Deserialize<T>(byte[] stream);
    }

public class ProtobufNetSerializer : ICacheDataSerializer
    {
        public byte[] Serialize(object o)
        {
            using (var memoryStream = new MemoryStream())
            {
                Serializer.Serialize(memoryStream, o);

                return memoryStream.ToArray();
            }
        }

        public T Deserialize<T>(byte[] stream)
        {
            var memoryStream = new MemoryStream(stream);

            return Serializer.Deserialize<T>(memoryStream);
        }
    }

实例化如下：

using Microsoft.Practices.TransientFaultHandling;

public class RedisCacheTransientErrorDetectionStrategy : ITransientErrorDetectionStrategy
    {
        /// <summary>
        /// Custom Redis Transient Error Detenction Strategy must have been implemented to satisfy Redis exceptions.
        /// </summary>
        /// <param name="ex"></param>
        /// <returns></returns>
        public bool IsTransient(Exception ex)
        {
            if (ex == null) return false;

            if (ex is TimeoutException) return true;

            if (ex is RedisServerException) return true;

            if (ex is RedisException) return true;

            if (ex.InnerException != null)
            {
                return IsTransient(ex.InnerException);
            }

            return false;
        }
    }

像这样使用：

private readonly RetryPolicy _retryPolicy;

// CODE
var retryStrategy = new FixedInterval(3, TimeSpan.FromSeconds(2));
            _retryPolicy = new RetryPolicy<RedisCacheTransientErrorDetectionStrategy>(retryStrategy);

查看代码，以最大限度地减少缓存中存储的缓存调用和值。我通过更有效地存储值来减少大量错误。

如果这些都没有帮助。移动到更高的缓存（我们最终使用C3而不是C1）。

enter image description here

答案 1 :(得分：1)

在所有异常消息中，您都有类似＆＃34; in = 65536/0＆＃34;的内容。这表示本地已收到65536个字节（在本地计算机的内核插槽缓冲区中），但尚未由StackExchange.Redis处理。这将指出客户端应用程序方面的问题，即使系统无法处理您的响应。我能想到的两种可能性可能导致这种情况：

  1. 您的系统没有足够的可用线程池线程来足够快地处理数据。如果更新为构建1.0.450或更高版本的StackExchange.Redis，则超时消息将包含有关线程池繁忙程度的统计信息，这将有助于诊断此问题。或者，您可以使用https://github.com/JonCole/SampleCode/blob/master/ThreadPoolMonitor/ThreadPoolLogger.cs之类的代码在收到异常时自行记录相同的信息。

  2. 您正在从缓存中接收一个非常大的对象，并且在StackExchange.Redis上配置的超时内未收到对象的大小。发生这种情况时，排在大项后面的同一连接上的所有调用都将在大项超时时超时。

    3. 无论如何，希望这有帮助。

答案 2 :(得分：0)

转到Azure并获取更大的Redis实例。那个

相关问题
最新问题