Question

我正在使用django + celery和redis作为代理，其中一项任务涉及读取大小约为25MB的大文件并返回结果，然后链接另一个任务来处理结果。

由于我对redis不熟悉，我在这里遇到错误，我正在寻求帮助。可能是什么问题？

[2013-06-23 22:45:41,241: ERROR/MainProcess] Unrecoverable error: MemoryError()
Traceback (most recent call last):
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/celery/worker/__init__.py", line 363, in start
    component.start()
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/celery/worker/consumer.py", line 395, in start
    self.consume_messages()
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/celery/worker/consumer.py", line 480, in consume_messages
    readers[fileno](fileno, event)
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/kombu/transport/redis.py", line 770, in handle_event
    self._callbacks[queue](message)
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/kombu/transport/virtual/__init__.py", line 479, in _callback
    self.qos.append(message, message.delivery_tag)
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/kombu/transport/redis.py", line 117, in append
    dumps([message._raw, EX, RK])) \
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/redis/client.py", line 1808, in execute
    return execute(conn, stack, raise_on_error)
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/redis/client.py", line 1705, in _execute_transaction
    [args for args, options in cmds]))
MemoryError

不确定它是否提供任何提示，但检查redis服务器上的maxmemory设置似乎不是问题。

redis 127.0.0.1:6379> config get maxmemory
1) "maxmemory"
2) "3758096384"

Answer 1

看起来内存错误不在redis端，而是在客户端（celery worker）

我的猜测是工人内存不足

你应该确保芹菜过程实际上可以将来自redis的结果分配到内存中。

如果在执行少量任务后发生这种情况，则意味着您没有足够的内存来处理您设置的并发性或者您正在泄漏内存（将对象的引用留在redis中）

django-celery redis memoryerror

1 个答案: