我试图让多处理ServerApp
在Windows上运行。我想这个问题缺少os.fork()
功能,所以我不得不以某种方式传递socket
不可拾取的内容(?!)。
我已经看到使用来自reduce_handle
的{{1}}和rebuild_handle
可能会出现here,但这些方法在Python 3中不可用(? !)。虽然我可以使用multiprocessing.reduction
和duplicate
,但我无法找到如何使用它们的示例,或者我是否需要它们。
此外,我想知道创建新流程时steal_handle
是否会成为问题?
这是我的ServerApp示例:
logging
答案 0 :(得分:3)
要允许python3的连接pickle(包括套接字),您应该使用mulitprocessing.allow_connection_pickling
。它在ForkingPickler
中为套接字注册Reducer。例如:
import socket
import multiprocessing as mp
mp.allow_connection_pickling()
def _test_connection(conn):
msg = conn.recv(2)
conn.send(msg)
conn.close()
print("ok")
if __name__ == '__main__':
server, client = socket.socketpair()
p = mp.Process(target=_test_connection, args=(server,))
p.start()
client.settimeout(5)
msg = b'42'
client.send(msg)
assert client.recv(2) == msg
p.join()
assert p.exitcode == 0
client.close()
server.close()
我还注意到你对socket
的腌制还有一些其他问题没有实现。
当使用self.conn_handler
作为目标时,多处理将尝试挑选整个对象self
。这是一个问题,因为您的对象包含一些无法腌制的Thread
。因此,您应该从目标函数的闭包中删除self
。可以使用@staticmethod
装饰器并删除函数中提及的所有self
来完成此操作。
此外,logging
模块不会处理多个进程。基本上,启动的Process
中的所有日志都将使用您当前的代码丢失。要解决此问题,您可以在开始第二个logging
(Process
开头)或使用conn_handler
日志记录实用程序时启动新的multiprocessing
。
这可以是这样的:
import logging
import socket
from select import select
from threading import Thread
from multiprocessing import util, get_context
from sys import stdout
from time import sleep
util.log_to_stderr(20)
ctx = get_context("spawn")
class ServerApp(object):
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
handler = logging.StreamHandler(stdout)
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
def __init__(self, id, port=4545, ip='127.0.0.1', method='tcp',
buffer_size=2048):
self.id = id
self.port = port
self.ip = ip
self.socket = None
self.listener = None
self.buffer_size = buffer_size
# Additional attributes here....
self.clients = []
self.client_buffers = []
@staticmethod
def conn_handler(id, connection, address, buffer):
print("test")
util.info("[%d] - Connection from %s:%d", id, address[0], address[1])
try:
while True:
command = None
received_data = b''
# Check for client commands
readable, writable, exceptional = select([connection], [], [],
0)
if readable:
# Get Command ... There is more code here
command = 'Something'
if command == 'Something':
connection.sendall(b"Coucouc")
break
else:
print(':(')
sleep(.1)
except Exception as e:
print(e)
finally:
connection.close()
util.info("[%d] - Connection from %s:%d has been closed.", id,
address[0], address[1])
print("Close")
def join(self):
while self.listener.is_alive():
self.listener.join(0.5)
def acceptor(self):
while True:
self.logger.info("[%d] - Waiting for connection on %s:%d", self.id,
self.ip, self.port)
# Accept a connection on the bound socket and fork a child process
# to handle it.
conn, address = self.socket.accept()
# Create Queue which will represent buffer for specific client and
# add it o list of all client buffers
buffer = ctx.Queue()
self.client_buffers.append(buffer)
process = ctx.Process(target=self.conn_handler,
args=(self.id, conn, address, buffer))
process.daemon = True
process.start()
self.clients.append(process)
# Close the connection fd in the parent, since the child process
# has its own reference.
conn.close()
def run(self):
# Create TCP socket, bind port and listen for incoming connections
self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.socket.bind((self.ip, self.port))
self.socket.listen(5)
# Run acceptor thread to handle new connection
self.listener = Thread(target=self.acceptor)
self.listener.daemon = True
self.listener.start()
self.listener.join()
def main():
app = ServerApp(0)
app.run()
if __name__ == '__main__':
main()
我只在Unix和python3.6上测试过,但它的行为不应该与我在windows中使用spawn context , which should behave like the
Process`时的行为不同。