如何避免Redis LockRelease调用

时间:2016-05-20 19:45:56

标签: redis stackexchange.redis azure-redis-cache

当我尝试释放redis锁时,我一直收到以下超时异常。我正在使用Azure Redis和StackExchange.Redis版本1.1.603。我得到的例外情况如下。我查看了异常错误中包含的链接,但我没有看到任何与此方案有关的内容。

Unexpected lock release failure on lock key 1ea47191-85e6-4f89-ad20-be700d643d6a-ProbeIdsLock on machine ELITEBOOK. Timeout performing EX
EC, inst: 0, mgr: Inactive, err: never, queue: 3, qu: 1, qs: 2, qc: 0, wr: 1, wq: 1, in: 101, ar: 0, clientName: ELITEBOOK, IOCP: (Busy=1,Free=
999,Min=4,Max=1000), WORKER: (Busy=1,Free=1022,Min=4,Max=1023), Local-CPU: 100% (Please take a look at this article for some common client-side
 issues that can cause timeouts: https://github.com/StackExchange/StackExchange.Redis/tree/master/Docs/Timeouts.md) -    at StackExchange.Redis
.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in C:\TeamCity\buildAgent\work\3
ae0647004edff78\StackExchange.Redis\StackExchange\Redis\ConnectionMultiplexer.cs:line 2044
         at StackExchange.Redis.RedisTransaction.Execute(CommandFlags flags) in C:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redi
s\StackExchange\Redis\RedisTransaction.cs:line 51
         at StackExchange.Redis.RedisDatabase.LockRelease(RedisKey key, RedisValue value, CommandFlags flags) in C:\TeamCity\buildAgent\work\3a
e0647004edff78\StackExchange.Redis\StackExchange\Redis\RedisDatabase.cs:line 838
         at ENow.RedisCache.RedisConnection.LockRelease(RedisKey key, RedisValue value, CommandFlags commandFlags)
         at ENow.RedisCache.MonitoringCache.LockRelease(RedisKey lockKey, RedisValue token)

代码基本上如下所示。 Redis调用被包装,以便最终编写单元测试更容易。

        public async Task<bool> AddProbeStatus(string tenantId, ProbeStatus probeStatus)
    {
        probeStatus.ProbeId.ThrowIfNullOrEmpty("probeStatus.ProbeId");
        var key = $"{tenantId}-ProbeIds";
        var lockKey = key + "Lock";
        RedisValue token = Environment.MachineName;
        var lockAquired = false;
        var finalResult = false;
        var retryCount = 0;
        try
        {
            while (!lockAquired)
            {
                retryCount++;
                lockAquired = _redisConnection.LockTake(lockKey, token, TimeSpan.MaxValue);

                if (lockAquired)
                {
                    _log.LogDebug($"Probe status lock aquired for tenant {tenantId}");

                    var result = await _redisConnection.StringGetAsync(key);

                    List<ProbeStatus> probeStatuses;
                    if (result == RedisValue.Null)
                    {
                        _log.LogDebug(
                            $"No probe status cache found for tenant {tenantId}. Adding probe {probeStatus.ProbeId} to cache.");
                        probeStatus.LastCheckinTimeInUtc = DateTime.UtcNow;
                        probeStatuses = new List<ProbeStatus> {probeStatus};
                    }
                    else
                    {
                        probeStatuses = JsonConvert.DeserializeObject<List<ProbeStatus>>(result);
                        var existingProbeStatus = probeStatuses.FirstOrDefault(p => p.ProbeId == probeStatus.ProbeId);
                        if (existingProbeStatus != null)
                        {
                            _log.LogDebug(
                                $"Updating last checkin time for probe {probeStatus.ProbeId} for tenant {tenantId} in probe status cache.");
                            existingProbeStatus.LastCheckinTimeInUtc = probeStatus.LastCheckinTimeInUtc;
                        }
                        else
                        {
                            _log.LogDebug(
                                $"Adding new probe {probeStatus.ProbeId} for tenant {tenantId} to probe status cache.");
                            probeStatuses.Add(probeStatus);
                        }
                    }
                    var probeStatusesAsJson = JsonConvert.SerializeObject(probeStatuses);
                    finalResult = await _redisConnection.StringSetAsync(key, probeStatusesAsJson);
                }
                else
                {
                    if (retryCount > MaxRetryCount)
                    {
                        var errorMessage =
                            $"Retry count exceeded while attempting to aquire lock in probe status cache for tenant {tenantId}";
                        _log.LogError(errorMessage);
                        throw new Exception(errorMessage);
                    }
                    _log.LogWarning($"Couldn't aquire lock in probe status cache for tenant {tenantId}. Waiting {DefaultRetrySeconds} seconds before retrying");
                    await Task.Delay(TimeSpan.FromSeconds(DefaultRetrySeconds));
                }
            }
        }
        catch (Exception ex)
        {
            _log.LogError($"Failed to set probe IDs for tenant {tenantId}. {ex.Message}");
            throw;
        }
        finally
        {
            if (lockAquired)
            {
                if(!LockRelease(lockKey, token))
                {
                    finalResult = false;
                }
            }
            else
            {
                _log.LogDebug($"Lock not aquired for key {lockKey} on machine {token}");
            }
        }
        return finalResult;
    }

此代码是从ASP.NET 5 RC1 WebAPI调用的。如果我在Azure或本地运行它,我会得到超时。但是,我有一个简单的XUnit集成测试,它可以运行相同的代码,并且在同一个Azure托管的Redis缓存中无异常运行。是否存在与ASP.NET 5 RC1和StackExchange.Redis版本1.1.603不兼容的问题?奇怪的是,这个问题在WebAPI中始终存在。它不是任何间歇性的。此外,它总是成功使用集成测试,所以感觉就像其他东西在这里,并且超时只是一个红色的鲱鱼。

0 个答案:

没有答案