我将celery库与Python一起用于并行处理一些相当大的数据集。但是我每天运行的cronjob每隔两三天就会中断一次,并产生以下错误。
CRITICAL/MainProcess] Unrecoverable error: ResponseError('MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. Commands that may modify the data set are disabled, because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option). Please check the Redis logs for details about the RDB error.',)
redis日志看起来像
7633:M 12 Apr 07:56:28.284 * 1 changes in 900 seconds. Saving...
7633:M 12 Apr 07:56:28.288 * Background saving started by pid 9530
9530:C 12 Apr 07:56:28.372 * DB saved on disk
9530:C 12 Apr 07:56:28.373 * RDB: 2 MB of memory used by copy-on-write
7633:M 12 Apr 07:56:28.388 * Background saving terminated with success
7633:M 12 Apr 08:01:29.065 * 10 changes in 300 seconds. Saving...
7633:M 12 Apr 08:01:29.065 # Can't save in background: fork: Cannot allocate memory
7633:M 12 Apr 08:01:35.083 * 10 changes in 300 seconds. Saving...
7633:M 12 Apr 08:01:35.088 * Background saving started by pid 9998
9998:C 12 Apr 08:01:35.099 * DB saved on disk
9998:C 12 Apr 08:01:35.101 * RDB: 2 MB of memory used by copy-on-write
7633:M 12 Apr 08:01:35.188 * Background saving terminated with success
相关的系统配置是
有趣的是,完全相同的配置和系统在另一台(开发)服务器上运行良好,并且在另一台运行Ubuntu 16.04的服务器上运行良好。但是生产失败了。请注意,我不是celery和redis方面的专家,只是经过大量的Google搜寻并遵循了一些教程之后,代码才能正常工作。因此,请尝试将故障排除步骤保持在相对基本的水平。另外,当我在外壳上ping redis-cli时,会得到一个pong作为回报,这表明服务器正在运行。