Postgres关闭了与芹菜工人的联系

时间:2016-01-12 11:58:43

标签: django postgresql heroku celery

我有一个在Heroku上托管的Django(1.8)应用程序并使用Postgres(9.4.4)数据库。有2个专业dynos:一个运行Web服务器,一个运行Celery(3.1)工作器,Celery配置database backend

应用程序在过去几周/几个月内一直运行顺畅但几天后我偶尔经历几秒钟的停机时间。

从日志中看,它似乎与因未知原因而被循环的dynos有关:

...<some successful requests>...
heroku[web.1]: State changed from up to starting
heroku[web.1]: Cycling
heroku[web.1]: Stopping all processes with SIGTERM
... Cycling in progress
heroku[web.1]: State changed from starting to up

根据heroku documentation,这可能是“OK”,因为dynos需要每天至少循环一次。

与此同时,工人也会遇到同样的事情,但这就是问题所在:

heroku[worker.1]: State changed from up to starting
heroku[worker.1]: Cycling
heroku[worker.1]: Stopping all processes with SIGTERM
app[worker.1]: worker: Warm shutdown (MainProcess)
app[worker.1]: [INFO/MainProcess] beat: Shutting down...

...Celery being restarted...

app[worker.1]: [INFO/Beat] beat: Starting...
app[worker.1]: [INFO/MainProcess] Connected to amqp://my-queue-address
app[worker.1]: [INFO/Beat] Writing entries...
app[worker.1]: [INFO/MainProcess] mingle: searching for neighbors
app[worker.1]: [WARNING/MainProcess] celery@xxxxxxxxxxx ready.
app[worker.1]: [INFO/MainProcess] mingle: all alone
app[worker.1]: [INFO/Beat] Scheduler: Sending due task celery.backend_cleanup (celery.backend_cleanup)
app[worker.1]: [INFO/MainProcess] Received task: celery.backend_cleanup[13f89001-08b0-434b-bc7e-372ffcfef521]
app[worker.1]: [INFO/Beat] Writing entries...
app[worker.1]: [INFO/MainProcess] Task celery.backend_cleanup[13f89001-08b0-434b-bc7e-372ffcfef521] succeeded in 0.00162791600451s: None
app[worker.1]: [ERROR/Beat] Database gave error:
OperationalError('terminating connection due to administrator command\nSSL connection has been closed unexpectedly\n',)

...Flood of error logs...

app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 218, in _set_autocommit
app[worker.1]:     six.reraise(dj_exc_type, dj_exc_value, traceback)
app[worker.1]: InterfaceError: connection already closed

现在每天出现的问题超过15次。

有谁知道为什么dynos会收到这个SIGTERM信号?另外,为什么循环会导致“OperationalError:由于管理员命令而终止连接”错误?即使我们远远低于heroku设定的限制,为什么Postgres会关闭它的连接?

顺便说一句,该应用程序在本地运行Postgres 9.3时没有任何问题。

0 个答案:

没有答案