Question

我正在尝试使用django + celery + djcelery + rabbitmq + memcached将同步任务卸载到其他服务器。

根据芹菜日志和芹菜花，异步任务被发送给工人并成功执行。但结果似乎永远不会回来。

Web服务器上运行的代码是：

def wait_task(task, *args):
    async_result = task.delay(*args)
    return async_result.get()

@shared_task(serializer="pickle")
def render_pdf(pdf_id):
    pdf = do_stuff()
    return pdf

pdf = wait_task(render_pdf, pdf_id)

我的BROKER_URL设置为librabbitmq://guest:guest@172.17.0.5:5672/，这可以通过芹菜日志和芹菜花输出来证明。

芹菜花中显示的截断结果让人联想到成功呈现的PDF。

/etc/rabbitmq/rabbitmq-env.conf是：

NODENAME=rabbit@localhost
NODE_IP_ADDRESS=0.0.0.0

不确定下一步该去哪儿。假设celery没有返回/回调给rabbitmq，或者rabbitmq没有返回/回调到原来的wait_task调用。

如果我在与代理相同的VM上运行Web服务器，那么一切正常，即使使用相同的BROKER_URL。在配置的某个地方，我想假设它在哪里回调，比如localhost / 127.0.0.1或其他东西？

更新1 ：

从启动到第一次共享任务完成的Celery日志输出：

[2016-02-02 23:54:07,786: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2016-02-02 23:54:07,792: DEBUG/MainProcess] | Worker: Building graph...
[2016-02-02 23:54:07,793: DEBUG/MainProcess] | Worker: New boot order: {StateDB, Timer, Hub, Queues (intra), Pool, Autoreloader, Beat, Autoscaler, Consumer}
[2016-02-02 23:54:07,796: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2016-02-02 23:54:07,797: DEBUG/MainProcess] | Consumer: Building graph...
[2016-02-02 23:54:07,799: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Mingle, Gossip, Tasks, Control, Agent, Heart, event loop}
[2016-02-02 23:54:07,806: DEBUG/MainProcess] | Worker: Starting Hub
[2016-02-02 23:54:07,807: DEBUG/MainProcess] ^-- substep ok
[2016-02-02 23:54:07,807: DEBUG/MainProcess] | Worker: Starting Pool
[2016-02-02 23:54:07,811: DEBUG/MainProcess] ^-- substep ok
[2016-02-02 23:54:07,813: DEBUG/MainProcess] | Worker: Starting Autoscaler
[2016-02-02 23:54:07,813: DEBUG/MainProcess] ^-- substep ok
[2016-02-02 23:54:07,814: DEBUG/MainProcess] | Worker: Starting Consumer
[2016-02-02 23:54:07,815: DEBUG/MainProcess] | Consumer: Starting Connection
[2016-02-02 23:54:07,824: INFO/MainProcess] Connected to amqp://guest:**@172.17.0.5:5672//
[2016-02-02 23:54:07,825: DEBUG/MainProcess] ^-- substep ok
[2016-02-02 23:54:07,826: DEBUG/MainProcess] | Consumer: Starting Events
[2016-02-02 23:54:07,836: DEBUG/MainProcess] ^-- substep ok
[2016-02-02 23:54:07,837: DEBUG/MainProcess] | Consumer: Starting Mingle
[2016-02-02 23:54:07,838: INFO/MainProcess] mingle: searching for neighbors
[2016-02-02 23:54:08,848: INFO/MainProcess] mingle: all alone
[2016-02-02 23:54:08,850: DEBUG/MainProcess] ^-- substep ok
[2016-02-02 23:54:08,851: DEBUG/MainProcess] | Consumer: Starting Gossip
[2016-02-02 23:54:08,855: DEBUG/MainProcess] ^-- substep ok
[2016-02-02 23:54:08,856: DEBUG/MainProcess] | Consumer: Starting Tasks
[2016-02-02 23:54:08,862: DEBUG/MainProcess] ^-- substep ok
[2016-02-02 23:54:08,863: DEBUG/MainProcess] | Consumer: Starting Control
[2016-02-02 23:54:08,867: DEBUG/MainProcess] ^-- substep ok
[2016-02-02 23:54:08,868: DEBUG/MainProcess] | Consumer: Starting Heart
[2016-02-02 23:54:08,870: DEBUG/MainProcess] ^-- substep ok
[2016-02-02 23:54:08,871: DEBUG/MainProcess] | Consumer: Starting event loop
[2016-02-02 23:54:08,873: DEBUG/MainProcess] /virtualenv/processing/local/lib/python2.7/site-packages/celery/fixups/django.py:265: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2016-02-02 23:54:08,875: DEBUG/MainProcess] worker-1@172.17.0.5 ready.
[2016-02-02 23:54:08,876: DEBUG/MainProcess] | Worker: Hub.register Autoscaler...
[2016-02-02 23:54:08,877: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2016-02-02 23:54:08,878: DEBUG/MainProcess] basic.qos: prefetch_count->1
[2016-02-02 23:54:13,263: INFO/MainProcess] Received task: tasks.render_pdf[149cc075-1d09-4547-86ee-57608c1df8aa]
[2016-02-02 23:54:13,265: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f08fccd7488> (args:('tasks.render_pdf', '149cc075-1d09-4547-86ee-57608c1df8aa', (u'en', 8583L), {}, {'utc': True, u'is_eager': False, 'chord': None, u'group': None, 'args': (u'en', 8583L), 'retries': 0, u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': 'default', u'exchange': 'default'}, 'expires': None, u'hostname': 'worker-1@172.17.0.5', 'task': 'tasks.render_pdf', 'callbacks': None, u'correlation_id': '149cc075-1d09-4547-86ee-57608c1df8aa', 'errbacks': None, 'timelimit': (None, None), 'taskset': None, 'kwargs': {}, 'eta': None, u'reply_to': '8502acac-0475-3f84-b542-03fdf4ea4d3d', 'id': '149cc075-1d09-4547-86ee-57608c1df8aa', u'headers': {}}) kwargs:{})
[2016-02-02 23:54:13,267: DEBUG/MainProcess] Task accepted: tasks.render_pdf[149cc075-1d09-4547-86ee-57608c1df8aa] pid:5033
[2016-02-02 23:54:14,193: INFO/MainProcess] Task tasks.render_pdf[149cc075-1d09-4547-86ee-57608c1df8aa] succeeded in 0.926254872s: '%PDF-1.4\n1 0 obj\n<<\n/Title (\xfe\xff\x00T\x00i\x00c\x00k\x00e\x00t\x00f\x00l\x00a\x00p\x00...

更新2 ：

缓存设置：

CELERY_RESULT_BACKEND = 'djcelery.backends.cache:CacheBackend'
MEMCACHED_HOST = '172.17.0.5'
MEMCACHED_PORT = 11211
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': '%s:%s' % (MEMCACHED_HOST, str(MEMCACHED_PORT)),
        'TIMEOUT': 18000  # 5 hours
    },
}

更新3 ：

/var/logs/rabbitmq/rabbitmq@localhost.log中的输出：

如果我在上述失败的单独实例上运行celery和web服务器：

=INFO REPORT==== 2-Feb-2016::16:12:33 ===
accepting AMQP connection <0.469.0> (172.17.0.6:36786 -> 172.17.0.5:5672)

如果我在同一个实例上运行芹菜和Web服务器，那么工作正常：

=INFO REPORT==== 2-Feb-2016::16:13:09 ===
accepting AMQP connection <0.482.0> (172.17.0.5:33863 -> 172.17.0.5:5672)

Answer 1

我以前的期望是结果将通过rabbitmq返回，而不是直接memcached。

公开端口11211并更改memcached配置以侦听0.0.0.0修复它：

根据{<1}}在-l设置/etc/memcached.conf：

# -l 127.0.0.1
-l 0.0.0.0

尽管成功，芹菜任务结果仍未返回

1 个答案: