我遇到了一个问题,在通过RabbitMq在MassTransit中发送消息时,偶尔会抛出LockRecursionException。
System.Threading.LockRecursionException:保持读锁定时可能无法获取写锁定。这种模式容易出现死锁
在类MassTransit.Transports.DefaultConnectionPolicy
中引发异常,看起来Execute
方法获取读锁定,运行回调并始终通过finally块释放读锁定。但是,无论出于何种原因,回调都会引发InvalidConnectionException,代码会尝试Reconnect()
。 Reconnect()
将尝试在TryEnterWriteLock中获取写锁定,并以System.Threading.LockRecursionException结束,因为当前线程已经保持读锁定。我没有看到,如果Finally块应该释放读锁定,那么可能会发生这种情况,除非ExitReadLock没有做它想做的事情。
我已经从MassTransit源代码和完整堆栈跟踪中包含了完整的类。
public class DefaultConnectionPolicy : ConnectionPolicy
{
readonly ConnectionHandler _connectionHandler;
readonly TimeSpan _reconnectDelay;
readonly ILog _log = Logger.Get(typeof(DefaultConnectionPolicy));
readonly ReaderWriterLockSlim _connectionLock = new ReaderWriterLockSlim(LockRecursionPolicy.SupportsRecursion);
public DefaultConnectionPolicy(ConnectionHandler connectionHandler)
{
_connectionHandler = connectionHandler;
_reconnectDelay = 10.Seconds();
}
public void Execute(Action callback)
{
try
{
try
{
// wait here so we can be sure that there is not a reconnect in progress
_connectionLock.EnterReadLock();
callback();
}
finally
{
_connectionLock.ExitReadLock();
}
}
catch (InvalidConnectionException ex)
{
_log.Warn("Invalid Connection when executing callback", ex.InnerException);
Reconnect();
if (_log.IsDebugEnabled)
{
_log.Debug("Retrying callback after reconnect.");
}
try
{
// wait here so we can be sure that there is not a reconnect in progress
_connectionLock.EnterReadLock();
callback();
}
finally
{
_connectionLock.ExitReadLock();
}
}
}
void Reconnect()
{
if (_connectionLock.TryEnterWriteLock((int)_reconnectDelay.TotalMilliseconds/2))
{
try
{
if (_log.IsDebugEnabled)
{
_log.Debug("Disconnecting connection handler.");
}
_connectionHandler.Disconnect();
if (_reconnectDelay > TimeSpan.Zero)
Thread.Sleep(_reconnectDelay);
if (_log.IsDebugEnabled)
{
_log.Debug("Re-connecting connection handler...");
}
_connectionHandler.Connect();
}
catch (Exception)
{
_log.Warn("Failed to reconnect, deferring to connection policy for reconnection");
_connectionHandler.ForceReconnect(_reconnectDelay);
}
finally
{
_connectionLock.ExitWriteLock();
}
}
else
{
try
{
_connectionLock.EnterReadLock();
if (_log.IsDebugEnabled)
{
_log.Debug("Waiting for reconnect in another thread.");
}
}
finally
{
_connectionLock.ExitReadLock();
}
}
}
}
完整的堆栈跟踪
An exception was thrown during Send ---> System.Threading.LockRecursionException: Write lock may not be acquired with read lock held. This pattern is prone to deadlocks. Please ensure that read locks are released before taking a write lock. If an upgrade is necessary, use an upgrade lock in place of the read lock.
at System.Threading.ReaderWriterLockSlim.TryEnterWriteLockCore(TimeoutTracker timeout)
at System.Threading.ReaderWriterLockSlim.TryEnterWriteLock(TimeoutTracker timeout)
at MassTransit.Transports.DefaultConnectionPolicy.Reconnect() in z:\Builds\work\4ed32a1c3fc3f594\src\MassTransit\Transports\DefaultConnectionPolicy.cs:line 75
at MassTransit.Transports.DefaultConnectionPolicy.Execute(Action callback) in z:\Builds\work\4ed32a1c3fc3f594\src\MassTransit\Transports\DefaultConnectionPolicy.cs:line 53
at MassTransit.Transports.ConnectionPolicyChainImpl.Next(Action callback) in z:\Builds\work\4ed32a1c3fc3f594\src\MassTransit\Transports\ConnectionPolicyChainImpl.cs:line 49
at MassTransit.Transports.ConnectionHandlerImpl`1.Use(Action`1 callback) in z:\Builds\work\4ed32a1c3fc3f594\src\MassTransit\Transports\ConnectionHandlerImpl.cs:line 86
at MassTransit.Transports.RabbitMq.OutboundRabbitMqTransport.Send(ISendContext context) in z:\Builds\work\4ed32a1c3fc3f594\src\Transports\MassTransit.Transports.RabbitMq\OutboundRabbitMqTransport.cs:line 51
at MassTransit.Transports.Transport.Send(ISendContext context) in z:\Builds\work\4ed32a1c3fc3f594\src\MassTransit\Transports\Transport.cs:line 50
at MassTransit.Transports.Endpoint.Send[T](ISendContext`1 context) in z:\Builds\work\4ed32a1c3fc3f594\src\MassTransit\Transports\Endpoint.cs:line 111
--- End of inner exception stack trace ---
at MassTransit.Transports.Endpoint.Send[T](ISendContext`1 context) in z:\Builds\work\4ed32a1c3fc3f594\src\MassTransit\Transports\Endpoint.cs:line 117
at MassTransit.Pipeline.Sinks.EndpointMessageSink`1.<Enumerate>b__0(IBusPublishContext`1 x) in z:\Builds\work\4ed32a1c3fc3f594\src\MassTransit\Pipeline\Sinks\EndpointMessageSink.cs:line 45
at MassTransit.Pipeline.Sinks.OutboundConvertMessageSink`1.<>c__DisplayClass2.<>c__DisplayClass4.<Enumerate>b__1(ISendContext x) in z:\Builds\work\4ed32a1c3fc3f594\src\MassTransit\Pipeline\Sinks\OutboundConvertMessageSink.cs:line 36
at MassTransit.ServiceBus.Publish[T](T message, Action`1 contextCallback) in z:\Builds\work\4ed32a1c3fc3f594\src\MassTransit\ServiceBus.cs:line 180
--- End of inner exception stack trace ---
at MassTransit.ServiceBus.Publish[T](T message, Action`1 contextCallback) in z:\Builds\work\4ed32a1c3fc3f594\src\MassTransit\ServiceBus.cs:line 207
at MassTransit.Context.BusObjectPublisherImpl`1.Publish(IServiceBus bus, Object message, Action`1 contextCallback) in z:\Builds\work\4ed32a1c3fc3f594\src\MassTransit\Context\BusObjectPublisherImpl.cs:line 30
at MassTransit.ServiceBus.Publish(Object message, Type messageType, Action`1 contextCallback) in z:\Builds\work\4ed32a1c3fc3f594\src\MassTransit\ServiceBus.cs:line 244
at CollectionHouse.MassTransit.Messaging.MassTransitMessageBus.PublishImmediate(Object message)
at CollectionHouse.MassTransit.Services.MassTransitPublisher.Publish(Object message)
at CollectionHouse.MicroBus.EventBroadcast.MessageBroadcastHandler.<Handle>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Enexure.MicroBus.PipelineBuilder.<>c__DisplayClass7_0.<<GenerateNext>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at CollectionHouse.MicroBus.PipelineHandlers.LoggingHandler.<Handle>d__3.MoveNext()