I'm implementing circuit breaker on our service. It's quite standard Circuit Breaker except the fact that it wraps all the actions in timeout so it also breaks the circuit when too many timeouts occur.
When Circuit Breaker turns Half-Open state I'm using MonitorTryEnter
method to ensure only one thread executes the action ( to check if service recovered ). Please find the snippet below ( this is a code executed during Half-Open state):
EDIT ( pasted whole function code )
public async Task<T> ExecuteWithCircuitBreaker<T>(Func<Task<T>> func)
{
if (!_circuitBreakerStateManager.IsInState(CircuitBreakerState.Closed))
{
if (HalfOpen())
{
bool lockTaken = false;
try
{
Monitor.TryEnter(_halfOpenSyncObject, ref lockTaken);
if (lockTaken)
{
var timeoutTask = Task.Delay(_options.ActionTimeout);
var resultTask = func.Invoke();
var firstFinishedTask = await Task.WhenAny(timeoutTask, resultTask);
if (firstFinishedTask == timeoutTask)
{
throw new TimeoutException($"Operation reached timeout configured for this circuit breaker. Timeout: {_options.ActionTimeout.TotalSeconds} s");
}
else
{
Reset();
return await resultTask;
}
}
}
catch (Exception ex)
{
Trip(ex);
throw;
}
finally
{
if (lockTaken)
{
Monitor.Exit(_halfOpenSyncObject);
}
}
}
throw new CircuitBreakerOpenException(_circuitBreakerStateManager.LastException);
}
// Circuit Breaker is closed - execute function normally
try
{
var timeoutTask = Task.Delay(_options.ActionTimeout);
var resultTask = func.Invoke();
var firstFinishedTask = await Task.WhenAny(timeoutTask, resultTask);
if (firstFinishedTask == timeoutTask)
throw new TimeoutException($"Operation reached timeout configured for this circuit breaker. Timeout: {_options.ActionTimeout.TotalSeconds} s");
else
return await resultTask;
}
catch (Exception ex)
{
Trip(ex);
throw;
}
}
As you can see in above snippet TryEnter is wrapped in Try/Finally which ensures lock will be released no matter what. The problem, I am having is that at some point Monitor.TryEnter
starts to return false for every attemp. I debugged the code and found that before it starts happening, for every Monitor.TryEnter
call, Monitor.Exit
is executed - meaning the lock should be released, but it seems not as at this point every call returns false. Not sure how this is possible.
Is it possible that Monitor.Exit
doesn't release the lock because of some reason ( it does not throw any exception though ) ? Or maybe i'm doing something wrong in here ?
EDIT. This is how _halfOpenSyncObject is declared ( there is only one instance of this class )
public class TimeoutableCircuitBreaker: ICircuitBreaker
{
private readonly object _halfOpenSyncObject = new object();
...
EDIT HalfOpen implementation:
private bool HalfOpen()
{
var isHalfOpen = _circuitBreakerStateManager.GetLastStateChangedDate() + _options.BreakDuration < DateTime.UtcNow;
if (isHalfOpen)
_circuitBreakerStateManager.SetState(CircuitBreakerState.HalfOpen);
return isHalfOpen;
}