Question

如here所述，我们可以通过将工人定义为HTTP API端点，或使用宿主语言编写客户端（使用来自代理的消息并对其采取行动）来创建不同语言的工作者。

有另一种方法可以通过使用jupyter_kernels来实现，我正在尝试实现这一目标。这是扩展工作人员蓝图的步骤从芹菜进口芹菜从芹菜导入启动步骤从jupyter_client导入MultiKernelManager

from tornado import gen
from zmq.eventloop import ioloop
from zmq.eventloop.zmqstream import ZMQStream
ioloop.install()

app = Celery('tasks', broker='redis://', backend='redis://')

reply_futures = {}

class KernelStep(bootsteps.StartStopStep):
    # Don't actually know on which step it should depend !!!
    requires = {'celery.worker.components:Pool'}

    def __init__(self, worker,**kwargs):
        worker.kernel_manager = MultiKernelManager()

    def start(self, worker):
        # Setting a kernel_client here for worker
        kid = worker.kernel_manager.start_kernel()
        kernel_client = worker.kernel_manager.get_kernel(kid).client()
        # initializing the zmq streams and attaching the callback to receive message
        # from the kernel
        shell_stream = ZMQStream(kernel_client.shell_channel.socket)
        iopub_stream = ZMQStream(kernel_client.iopub_channel.socket)
        shell_stream.on_recv_stream(partial(reply_callback, kernel_client.session))
        iopub_stream.on_recv_stream(partial(reply_callback, kernel_client.session))
        worker.kernel_client = kernel_client

    def stop(self, worker):
        pass

    def terminate(self, worker):
        worker.kernel_manager.shutdown_all()

    # defining the callback which will be executed when message arrives on
    # zmq stream
    @staticmethod
    def reply_callback(cls,session, stream, msg_list):
        idents, msg_parts = session.feed_identities(msg_list)
        reply = session.deserialize(msg_parts)
        parent_id = reply['parent_header'].get('msg_id')
        reply_future = reply_futures.get(parent_id)
        if reply_future:
            reply_future.set_result(reply)

app.steps['worker'].add(KernelStep)

任务的定义方式是，它应获得工作人员的kernel_client对象并使用它执行传递给它的代码。

@app.task
def pythontask(code):
    # I don't know how to get the kernel_client for current celery worker !!?
    kernel_client = self.get_current_worker().kernel_client

    # create a IOLoop
    loop = ioloop.IOLoop.current()
    # listen on the streams 
    # any way to leverage here the event loop of worker ???
    msg_id = loop.run_sync(lambda: execute(kernel_client,code))
    reply_msgs[msg_id] = []

    @gen.coroutine
    def execute(kernel_client, code):
        msg_id = kernel_client.execute(code)
        f = reply_futures[msg_id] = Future()
        yield f
        raise gen.Return(msg_id)

在这里我找不到任何方法可以在python任务中获取工作人员的kernel_client对象！！此外，是否有任何方法可以利用worker的事件循环来侦听语言内核发送的套接字中的消息？

芹菜和枣树的多语言工作者

0 个答案: