Question

我正在做的是，假设您有几个需要执行的工作流程。这些工作流程具有任务，任务的目标是不同的主机。最快的方法是运行流程中的每个工作流，并并行运行。

我正在尝试运行python多处理来执行我在celery的帮助下调用的远程函数。如果我只运行一个进程，我的程序运行正常。但是，当我运行多个进程时，我会收到以下错误。据我所知，问题是在同一频道上同时发布。不应在线程/等之间共享通道。

如何让Celery解决这个问题？这是一个我应该推出的参数＆＃39; celeryd＆＃39;命令，或者我需要在我的python程序中执行它？

    Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "testHello.py", line 16, in test_hello_aux
    print output.get()
  File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 169, in get
    no_ack=no_ack,
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 155, in wait_for
    on_interval=on_interval)
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 229, in consume
    no_ack=no_ack, accept=self.accept) as consumer:
  File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 359, in __init__
    self.revive(self.channel)
  File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 371, in revive
    self.declare()
  File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 381, in declare
    queue.declare()
  File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 505, in declare
    self.queue_declare(nowait, passive=False)
  File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 531, in queue_declare
    nowait=nowait)
  File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 1254, in queue_declare
    self._send_method((50, 10), args)
  File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 56, in _send_method
    self.channel_id, method_sig, args, content,
  File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 221, in write_method
    write_frame(1, channel, payload)
  File "/usr/local/lib/python2.7/dist-packages/amqp/transport.py", line 177, in write_frame
    frame_type, channel, size, payload, 0xce,
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 32] Broken pipe
Process Process-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "testHello.py", line 16, in test_hello_aux
    print output.get()
  File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 169, in get
    no_ack=no_ack,
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 155, in wait_for
    on_interval=on_interval)
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 229, in consume
    no_ack=no_ack, accept=self.accept) as consumer:
  File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 359, in __init__
Process Process-3:
    self.revive(self.channel)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 371, in revive
    self.declare()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 381, in declare
    queue.declare()
  File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 504, in declare
    self.run()
    self.exchange.declare(nowait)
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
  File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare
    self._target(*self._args, **self._kwargs)
    nowait=nowait, passive=passive,
  File "testHello.py", line 16, in test_hello_aux
  File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 613, in exchange_declare
    print output.get()
  File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 169, in get
    no_ack=no_ack,
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 155, in wait_for
    on_interval=on_interval)
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 229, in consume
    no_ack=no_ack, accept=self.accept) as consumer:
  File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 359, in __init__
    self._send_method((40, 10), args)
  File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 56, in _send_method
    self.channel_id, method_sig, args, content,
  File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 221, in write_method
    self.revive(self.channel)
  File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 371, in revive
    self.declare()
  File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 381, in declare
    write_frame(1, channel, payload)
    queue.declare()
  File "/usr/local/lib/python2.7/dist-packages/amqp/transport.py", line 177, in write_frame
  File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 504, in declare
    frame_type, channel, size, payload, 0xce,
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    self.exchange.declare(nowait)
  File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare
    nowait=nowait, passive=passive,
  File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 620, in exchange_declare
    return getattr(self._sock,name)(*args)
error: [Errno 32] Broken pipe
    (40, 11),  # Channel.exchange_declare_ok
  File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 67, in wait
    self.channel_id, allowed_methods)
  File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 237, in _wait_method
    self.method_reader.read_method()
  File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 189, in read_method
    raise m
error: [Errno 104] Connection reset by peer
Process Process-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "testHello.py", line 16, in test_hello_aux
    print output.get()
  File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 169, in get
    no_ack=no_ack,
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 155, in wait_for
    on_interval=on_interval)
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 229, in consume
    no_ack=no_ack, accept=self.accept) as consumer:
  File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 359, in __init__
    self.revive(self.channel)
  File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 371, in revive
    self.declare()
  File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 381, in declare
    queue.declare()
  File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 505, in declare
    self.queue_declare(nowait, passive=False)
  File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 531, in queue_declare
    nowait=nowait)
  File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 1258, in queue_declare
    (50, 11),  # Channel.queue_declare_ok
  File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 67, in wait
    self.channel_id, allowed_methods)
  File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 270, in _wait_method
    self.wait()
  File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 69, in wait
    return self.dispatch_method(method_sig, args, content)
  File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 87, in dispatch_method
    return amqp_method(self, args)
  File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 526, in _close
    (class_id, method_id), ConnectionError)
UnexpectedFrame: Basic.publish: (505) UNEXPECTED_FRAME - expected content header for class 60, got non content header frame instead

celery --version 3.1.11（Cipater） amq --version 0.9.1

Answer 1

使用Celery时，您不需要使用python多处理模块。芹菜为您提供一切照顾。

在名为tasks.py

的文件中定义您的任务


from celery import Celery
app = Celery('tasks', broker='amqp://guest@localhost//')

@app.task
def add(x, y):
    return x + y

现在假设add函数实际上是您希望并行运行的功能。我们也考虑一下术语。并行意味着同时，而异步意味着不同步。我无法保证您的任务将同时运行，但我可以保证它们不会同步运行。出于这个原因，让我们坚持使用术语async。

Celery有Canvas，一组用于异步流控制的基元。您感兴趣的两个是group和chord。 group允许您运行一组异步任务并询问阻止所有异步任务的结果（完成您尝试加入的内容）。 chord提供与group相同的功能，但在所有任务完成时触发回调。

调用代码的示例：



WAIT_TIME = 10 # how ever long you are willing to wait for your tasks

from tasks import add
from celery import group

future = group(add.s(i**i, i**i) for i in xrange(10))()
results = future.get(timeout=WAIT_TIME)

Celery任务会自动在他们自己的进程中运行（您生成的工作者），并且不需要您自己创建更多进程。

UNEXPECTED_FRAME - 类60的预期内容标题，取而代之的是非内容标题框

1 个答案: