使用pyzmq

Question

我尝试与通过ZeroMQ套接字开始使用multiprocessing.Process的子进程进行通信。我知道存在与multiprocessing模块中的子进程通信的解决方案，但我希望最终与来自用C ++编写的共享库中的函数进行通信。不用多说，这里是自包含的代码：

import time
import zmq
import multiprocessing

def perform(nseconds, endpoint):
    context = zmq.Context()
    publisher = context.socket(zmq.PUB)
    publisher.connect(endpoint)

    for i in range(5):
        time.sleep(nseconds)
        publisher.send_string("{}".format(i))
    publisher.send_string(">>END")

if __name__ == "__main__":
    multiprocessing.freeze_support()
    context = zmq.Context()
    socket = context.socket(zmq.SUB)
    socket.bind("tcp://*:*")
    socket.setsockopt_string(zmq.SUBSCRIBE, u"")
    endpoint = socket.getsockopt_string(zmq.LAST_ENDPOINT)

    print("Binding via {}".format(endpoint))

    t = multiprocessing.Process(target=perform, args=(1,endpoint))
    t.start()

    string = ""
    while not ">>END" in string:
        string = socket.recv_string()
        print(string)

    t.join()

此代码在GNU / Linux上运行时具有预期的输出：

Binding via tcp://0.0.0.0:34149
0
1
2
3
4
>>END

但是在Windows上使用来自Anaconda的Python和pyzmq版本16.0.2与conda install pyzmq一起安装时，它崩溃并出现以下错误：

Binding via tcp://0.0.0.0:52019
Assertion failed: Can't assign requested address (bundled\zeromq\src\tcp_connect
er.cpp:341)

我该如何解决这个问题？或者我做错了吗？如果我做错了，为什么它依赖于平台呢？

Answer 1

Remove the wildcard in socket.bind( "tcp://*:*" )

First,
this is very platform specific, one ought never rely on how the wildcard expansion will get handled on unknown ecosystem details in production. Be explicit.

Next,
this is on the very contrary to the concise, state-of-art resources management practice in Distributed-systems design and there could be hardly a worse idea from the architecture point of view, than to .bind() straight on all ports on all localhost available addresses. Just imagine what this causes inside the Context()-instance, to manage all that herd of endpoints and their associated resources' pools ready and sniffing for a potentially incoming connection request ( just in case one such may appear ). No.

Never do this.

---

Performance sins and other aspects _{- ref. my criticism on Amdahl's Law v/s costs}

Well, having seen your publications, I will not expand this a lot, but yet have to add, the Distributed-computing realm requires a lot of care spent on details -- never expect a few SLOCs to perform any good for HPC computations.

The worst sins come from a good will to improve some process. If a code spins-off a process ( what the multiprocessing is exactly designed to do ), not many people also associate with such a SLOC the actual costs, that have to be paid, before the departed code gets it's first chance to start processing ( replica of the complete python execution environment ( so as to escape from the original local-GIL etc ) -- so your code has to pay in both [TIME] and [SPACE] for a huge memory-to-memory transfer, next your proposed code instantiates another "remote"-process Context()-engine. While this may look smart, your code has to pay for it again all costs - and spend just a few SLOCs to process. Next, always use rather a .setsockopt( zmq.LINGER, 0 ), so as not to leave your resources infinitely blocking a graceful termination. While this seems as a "nice-to-have" preventive step, it is rather a "must-do" lifesaver, before hunting "what went wrong this time?" on a big and expensive computing infrastructure...

Many other performance tweaking deserves to be done before shelling out multiprocessing.Process( ... ), which goes way beyond the scope of this post. But worth mastering, definitely before running many-times inefficient code.