我有一个在Heroku上托管的Django(1.8)应用程序并使用Postgres(9.4.4)数据库。有2个专业dynos:一个运行Web服务器,一个运行Celery(3.1)工作器,Celery配置database backend。
应用程序在过去几周/几个月内一直运行顺畅但几天后我偶尔经历几秒钟的停机时间。
从日志中看,它似乎与因未知原因而被循环的dynos有关:
...<some successful requests>...
heroku[web.1]: State changed from up to starting
heroku[web.1]: Cycling
heroku[web.1]: Stopping all processes with SIGTERM
... Cycling in progress
heroku[web.1]: State changed from starting to up
根据heroku documentation,这可能是“OK”,因为dynos需要每天至少循环一次。
与此同时,工人也会遇到同样的事情,但这就是问题所在:
heroku[worker.1]: State changed from up to starting
heroku[worker.1]: Cycling
heroku[worker.1]: Stopping all processes with SIGTERM
app[worker.1]: worker: Warm shutdown (MainProcess)
app[worker.1]: [INFO/MainProcess] beat: Shutting down...
...Celery being restarted...
app[worker.1]: [INFO/Beat] beat: Starting...
app[worker.1]: [INFO/MainProcess] Connected to amqp://my-queue-address
app[worker.1]: [INFO/Beat] Writing entries...
app[worker.1]: [INFO/MainProcess] mingle: searching for neighbors
app[worker.1]: [WARNING/MainProcess] celery@xxxxxxxxxxx ready.
app[worker.1]: [INFO/MainProcess] mingle: all alone
app[worker.1]: [INFO/Beat] Scheduler: Sending due task celery.backend_cleanup (celery.backend_cleanup)
app[worker.1]: [INFO/MainProcess] Received task: celery.backend_cleanup[13f89001-08b0-434b-bc7e-372ffcfef521]
app[worker.1]: [INFO/Beat] Writing entries...
app[worker.1]: [INFO/MainProcess] Task celery.backend_cleanup[13f89001-08b0-434b-bc7e-372ffcfef521] succeeded in 0.00162791600451s: None
app[worker.1]: [ERROR/Beat] Database gave error:
OperationalError('terminating connection due to administrator command\nSSL connection has been closed unexpectedly\n',)
...Flood of error logs...
app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 218, in _set_autocommit
app[worker.1]: six.reraise(dj_exc_type, dj_exc_value, traceback)
app[worker.1]: InterfaceError: connection already closed
现在每天出现的问题超过15次。
有谁知道为什么dynos会收到这个SIGTERM信号?另外,为什么循环会导致“OperationalError:由于管理员命令而终止连接”错误?即使我们远远低于heroku设定的限制,为什么Postgres会关闭它的连接?
顺便说一句,该应用程序在本地运行Postgres 9.3时没有任何问题。