你如何处理芹菜(不是你的代码)引发的异常?

时间:2017-07-09 18:08:59

标签: python celery

所以现在我在我的烧瓶应用程序中使用Celery在远程计算机上部署服务器。现在,我有一个枚举状态,它表示我的部署过程的生命周期:

@celery.task(bind=True)
def deploy_server(self, server_id):
  server = Server.query.get(server_id)
  if not server.can_launch():
    return

  try:
    server.status = RemoteStatus.LAUNCHING
    db.session.commit()

    verify_DNS(server)

    host = server.server.ssh_user + '@' + server.server.ip
    execute(fabric_deploy_server, self, server, hosts=host)

    server.status = RemoteStatus.LAUNCHED
    db.session.commit()
  except Exception as e:
    server.status = RemoteStatus.ERROR
    db.session.commit()
    traceback.print_exc()
    raise e

如您所见,在部署服务器时,其状态将更改为"启动"。如果有异常,则会更改为ERROR。

我发现了一个完全绕过这个代码块的异常:当我用太多请求重载我的芹菜服务器时,我得到了这个例外:

[2017-07-09 18:00:03,127: WARNING/PoolWorker-3] /app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py:542: RuntimeWarning: Exception raised outside body: ConnectionError('max number of clients reached',): 
Traceback (most recent call last): 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 427, in trace_task 
    uuid, retval, task_request, publish_result, 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 152, in mark_as_done 
    self.store_result(task_id, result, state, request=request) 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 309, in store_result 
    request=request, **kwargs) 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 652, in _store_result 
    self.set(self.get_key_for_task(task_id), self.encode(meta)) 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 204, in set 
    return self.ensure(self._set, (key, value), **retry_policy) 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 194, in ensure 
    **retry_policy) 
  File "/app/.heroku/python/lib/python2.7/site-packages/kombu/utils/functional.py", line 333, in retry_over_time 
    return fun(*args, **kwargs) 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 213, in _set 
    pipe.execute() 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 2641, in execute 
    return execute(conn, stack, raise_on_error) 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 2495, in _execute_transaction 
    connection.send_packed_command(all_cmds) 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 538, in send_packed_command 
    self.connect() 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 446, in connect 
    self.on_connect() 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 514, in on_connect 
    if nativestr(self.read_response()) != 'OK': 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 577, in read_response 
    response = self._parser.read_response() 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 255, in read_response 
    raise error 
ConnectionError: max number of clients reached 
  exc, exc_info.traceback))) 

我最大的问题是这个错误是在我的Try / Catch集团之外的某个地方引发的。因此,当发生此异常时,我的所有服务器都保留在"启动"模式而不是"错误"。

如何捕获此异常并修改Server.status?

1 个答案:

答案 0 :(得分:0)

在redis 2.4中,存在最大连接数的硬编码限制,即10,000。在redis 2.6+中,您可以在redis.conf中指定最大客户端数。这也不是芹菜的问题,你的经纪人redis拒绝接受这个问题的联系。

使用redis CLI设置redis可以同时处理的最大客户端数。查看redis clients