Question

我试图让多处理ServerApp在Windows上运行。我想这个问题缺少os.fork()功能，所以我不得不以某种方式传递socket不可拾取的内容（？！）。

我已经看到使用来自reduce_handle的{{1}}和rebuild_handle可能会出现here，但这些方法在Python 3中不可用（？！）。虽然我可以使用multiprocessing.reduction和duplicate，但我无法找到如何使用它们的示例，或者我是否需要它们。

此外，我想知道创建新流程时steal_handle是否会成为问题？

这是我的ServerApp示例：

logging

Answer 1

要允许python3的连接pickle（包括套接字），您应该使用mulitprocessing.allow_connection_pickling。它在ForkingPickler中为套接字注册Reducer。例如：

import socket
import multiprocessing as mp
mp.allow_connection_pickling()


def _test_connection(conn):
    msg = conn.recv(2)
    conn.send(msg)
    conn.close()
    print("ok")

if __name__ == '__main__':
    server, client = socket.socketpair()

    p = mp.Process(target=_test_connection, args=(server,))
    p.start()

    client.settimeout(5)

    msg = b'42'
    client.send(msg)
    assert client.recv(2) == msg

    p.join()
    assert p.exitcode == 0

    client.close()
    server.close()

我还注意到你对socket的腌制还有一些其他问题没有实现。

当使用self.conn_handler作为目标时，多处理将尝试挑选整个对象self。这是一个问题，因为您的对象包含一些无法腌制的Thread。因此，您应该从目标函数的闭包中删除self。可以使用@staticmethod装饰器并删除函数中提及的所有self来完成此操作。
此外，logging模块不会处理多个进程。基本上，启动的Process中的所有日志都将使用您当前的代码丢失。要解决此问题，您可以在开始第二个logging（Process开头）或使用conn_handler日志记录实用程序时启动新的multiprocessing。

这可以是这样的：

import logging
import socket

from select import select
from threading import Thread
from multiprocessing import util, get_context
from sys import stdout
from time import sleep

util.log_to_stderr(20)
ctx = get_context("spawn")


class ServerApp(object):

    logger = logging.getLogger(__name__)
    logger.setLevel(logging.DEBUG)
    handler = logging.StreamHandler(stdout)
    formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
    handler.setFormatter(formatter)
    logger.addHandler(handler)

    def __init__(self, id, port=4545, ip='127.0.0.1', method='tcp',
                buffer_size=2048):

        self.id = id
        self.port = port
        self.ip = ip

        self.socket = None
        self.listener = None
        self.buffer_size = buffer_size

        # Additional attributes here....

        self.clients = []
        self.client_buffers = []

    @staticmethod
    def conn_handler(id, connection, address, buffer):

        print("test")
        util.info("[%d] - Connection from %s:%d", id, address[0], address[1])

        try:
            while True:

                command = None
                received_data = b''
                # Check for client commands
                readable, writable, exceptional = select([connection], [], [],
                                                        0)

                if readable:
                    # Get Command  ... There is more code here
                    command = 'Something'

                if command == 'Something':
                    connection.sendall(b"Coucouc")
                    break
                else:
                    print(':(')
                sleep(.1)

        except Exception as e:
            print(e)
        finally:
            connection.close()
            util.info("[%d] - Connection from %s:%d has been closed.", id,
                    address[0], address[1])
            print("Close")

    def join(self):

        while self.listener.is_alive():
            self.listener.join(0.5)

    def acceptor(self):

        while True:
            self.logger.info("[%d] - Waiting for connection on %s:%d", self.id,
                            self.ip, self.port)

            # Accept a connection on the bound socket and fork a child process
            # to handle it.
            conn, address = self.socket.accept()

            # Create Queue which will represent buffer for specific client and
            # add it o list of all client buffers
            buffer = ctx.Queue()
            self.client_buffers.append(buffer)

            process = ctx.Process(target=self.conn_handler,
                                args=(self.id, conn, address, buffer))
            process.daemon = True
            process.start()
            self.clients.append(process)

            # Close the connection fd in the parent, since the child process
            # has its own reference.
            conn.close()

    def run(self):

        # Create TCP socket, bind port and listen for incoming connections
        self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self.socket.bind((self.ip, self.port))
        self.socket.listen(5)

        # Run acceptor thread to handle new connection
        self.listener = Thread(target=self.acceptor)
        self.listener.daemon = True
        self.listener.start()

        self.listener.join()


def main():
    app = ServerApp(0)
    app.run()


if __name__ == '__main__':
    main()

我只在Unix和python3.6上测试过，但它的行为不应该与我在windows中使用spawn context , which should behave like the Process`时的行为不同。

Python3 Windows多处理将套接字传递给进程

1 个答案: