0mq一对多连接

时间:2011-08-24 15:45:58

标签: python multithreading zeromq

使用0mq在进程之间建立双向通信的最正确方法是什么?我需要创建几个后台进程,它们将等待来自主进程的命令,执行一些计算并将结果返回给主进程。

2 个答案:

答案 0 :(得分:7)

有几种方法可以做到这一点。最直接的方法可能是使用REQ / REP套接字。每个后台进程/ worker都有一个REP套接字,您可以使用REQ套接字与它们进行通信:

import zmq

def worker(addr):
    context = zmq.Context()
    socket = context.socket(zmq.REP)
    socket.bind(addr)
    while True:
        # get message from boss
        msg = socket.recv()
        # ...do smth
        # send back results
        socket.send(msg)

if __name__ == '__main__':
    # spawn 5 workers
    from multiprocessing import Process
    for i in range(5):
        Process(target=worker, args=('tcp://127.0.0.1:500%d' % i,)).start()

您必须连接到每个工作人员以向他们发送消息,并获取结果:

context = zmq.Context()
socket = context.socket(zmq.REQ)
socket.connect(worker_addr)
socket.send('message')
msg = socket.recv()

另一种方法是使用PUB / SUB向工作人员发送消息,并PUSH / PULL收集结果:

import zmq

def worker(worker_id, publisher_addr, results_addr):
    context = zmq.Context()
    sub = context.socket(zmq.SUB)
    sub.connect(publisher_addr)
    sub.setsockopt(zmq.SUBSCRIBE, worker_id)
    push = context.socket(zmq.PUSH)
    push.connect(results_addr)

    while True:
        msg = sub.recv_multipart()[1]
        # do smth, send off results
        push.send_multipart([worker_id, msg])

if __name__ == '__main__':
    publisher_addr = 'tcp://127.0.0.1:5000'
    results_addr = 'tcp://127.0.0.1:5001'

    # launch some workers into space
    from multiprocessing import Process
    for i in range(5):
        Process(target=worker, args=('worker-%d' % i, publisher_addr, results_addr,)).start()

要向特定工作人员广播命令,您可以执行以下操作:

context = zmq.Context()
pub = context.socket(zmq.PUB)
pub.bind(publisher_addr)
# send message to worker-1
pub.send_multipart(['worker-1', 'hello'])

拉入结果:

context = zmq.Context()
pull = context.socket(zmq.PULL)
pull.bind(results_addr)

while True:
    worker_id, result = pull.recv_multipart()
    print worker_id, result

答案 1 :(得分:3)

考虑使用Request Reply Broker,但将REQ套接字交换为DEALER。经销商不会阻止发送,并会自动为您的员工负载均衡流量。

在图片Client中,您的main processService A/B/C将是您的background processes (workers)Main process应绑定到端点。 Workers应连接到主进程的端点以接收工作项。

main process保留工作项列表并发送时间。如果一段时间内没有答案,则再次重新发送工作项,因为worker可能已经死亡。

Request Reply Broker