Question

我在我的一位Celery工作人员中收到以下错误：

2015-07-21T15:02:04.010066+00:00 app[worker.1]: Traceback (most recent call last):
2015-07-21T15:02:04.010069+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 296, in trace_task
2015-07-21T15:02:04.010070+00:00 app[worker.1]:     on_chord_part_return(task, state, R)
2015-07-21T15:02:04.010073+00:00 app[worker.1]:     deps.delete()
2015-07-21T15:02:04.010074+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/celery/result.py", line 773, in delete
2015-07-21T15:02:04.010071+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 587, in on_chord_part_return
2015-07-21T15:02:04.010078+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 329, in delete_group
2015-07-21T15:02:04.010076+00:00 app[worker.1]:     (backend or self.app.backend).delete_group(self.id)
2015-07-21T15:02:04.010079+00:00 app[worker.1]:     return self._delete_group(group_id)
2015-07-21T15:02:04.010081+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 499, in _delete_group
2015-07-21T15:02:04.010082+00:00 app[worker.1]:     self.delete(self.get_key_for_group(group_id))
2015-07-21T15:02:04.010083+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 172, in delete
2015-07-21T15:02:04.010084+00:00 app[worker.1]:     self.client.delete(key)
2015-07-21T15:02:04.010085+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 824, in delete
2015-07-21T15:02:04.010087+00:00 app[worker.1]:     return self.execute_command('DEL', *names)
2015-07-21T15:02:04.010088+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 565, in execute_command
2015-07-21T15:02:04.010089+00:00 app[worker.1]:     return self.parse_response(connection, command_name, **options)
2015-07-21T15:02:04.010090+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 579, in parse_response
2015-07-21T15:02:04.010091+00:00 app[worker.1]:     return self.response_callbacks[command_name](response, **options)
2015-07-21T15:02:04.010093+00:00 app[worker.1]: ValueError: invalid literal for int() with base 10: 'QUEUED'

我觉得奇怪的是我在堆栈跟踪的最后一行看不到int的调用。 QUEUED可能是作为工人的身份而来的。我将它用作这样的自定义工作者状态：

@before_task_publish.connect
def update_sent_state(sender=None, body=None, **kwargs):
    # the task may not exist if sent using `send_task` which
    # sends tasks by name, so fall back to the default result backend
    # if that is the case.
    task = current_app.tasks.get(sender)
    backend = task.backend if task else current_app.backend
    logging.debug("Setting status for %s" % body["id"])

    backend.store_result(body['id'], None, "QUEUED")

这可能是什么问题？

如果相关，请参阅我的任务代码。我只能直接调用fetch fetch。

@app.task
def fetch(url_or_urls, subscribe=None):
    """This fetches a (list of) podcast(s) and stores it in the db. It assumes that it only gets called
    by Podcast.get_by_url, or some other method that knows whether a given podcast has
    already been fetched.

    If *subscribe* is given, it should be a User instance to be subscribed to the given podcasts."""
    if isinstance(url_or_urls, basestring):
        url_or_urls = [url_or_urls]
    body = _store_podcasts.s()
    if subscribe:
        body.link(_subscribe_user.s(user=subscribe))
    return chord([_fetch_podcast_data.s(url) for url in url_or_urls])(body)

@app.task
def _fetch_podcast_data(url):
    return do_fetch(url) # This function returns a dict of podcast data.

@app.task
def _store_podcasts(podcasts_data):
    """Given a list of dictionaries representing podcasts, store them all in the database."""
    podcasts = [Podcast(**pdata) for pdata in podcasts_data]
    return Podcast.objects.insert(podcasts)

@app.task
def _subscribe_user(podcasts, user):
    """Subscribe the given users to all the podcasts in the list."""
    return user.subscribe_multi(podcasts)

这里还有其他相关内容吗？

pip freeze所示的图书馆版本：

redis==2.10.3
celery==3.1.18

Answer 1

如果没有工作代码，很难调试这样的错误。这就是我认为的可能。让我们从这里开始：

http://celery.readthedocs.org/en/latest/_modules/celery/backends/base.html#BaseBackend.store_result

def store_result(self, task_id, result, status,
                 traceback=None, request=None, **kwargs):
    """Update task state and result."""
    result = self.encode_result(result, status)
    self._store_result(task_id, result, status, traceback,
                       request=request, **kwargs)
    return result

它调用ecnode_result。让我们检查一下

  def encode_result(self, result, status):
        if status in self.EXCEPTION_STATES and isinstance(result, Exception):
            return self.prepare_exception(result)
        else:
            return self.prepare_value(result)

看起来像＃34;状态＆＃34;应该是预定义的STATE常量。

它的代码在这里

http://celery.readthedocs.org/en/latest/_modules/celery/states.html#state

这里有文档

http://celery.readthedocs.org/en/latest/reference/celery.states.html

看起来他们不希望看到像＆＃34; QUEUED＆＃34;那里。尝试其中一个预定义的。

Answer 2

redis python包期望DEL操作的响应始终为整数，我假设是已删除行的计数。

对int的调用发生在return self.response_callbacks[command_name](response, **options)等于self.response_callbacks['DEL']的最后一行（int）。

作为一种解决方法，您可以继承redis.client.StrictRedis并将DEL响应回调设置为int以外的其他回调，只需确保您熟悉其含义。< / p>

Answer 3

这些天我得到了同样的错误。并创建了我的QUEUED响应来自redis MULTI命令。请参阅https://redis.io/topics/transactions#usage。

也许您正在阅读错误连接的回复。也许在多处理/多线程/ eventlet等中不确定。

Redis和Celery的奇怪错误

3 个答案: