Question

我有一个应用程序同时使用grequests和multiprocessing.managers进行IPC通信和HTTP上的异步RESTful通信。

grequests使用gevent.monkey的{{1}}方法似乎打破了patch_all()类及其派生所使用的multiprocessing.connection模块。< / p>

这显然不是一个孤立的问题，但会影响任何实现multiprocessing.manager.SyncManager的用例，例如multiprocessing.connetion。

深入研究multiprocessing.pool中的代码，我发现stdlib gevent/monkey.py模块与socket的交换是造成破坏的原因。这可以在gevent.socket函数下的gevent/monkey.py的第115行找到：

patch_socket()

我的问题是为什么这个swappage中断def patch_socket(dns=True, aggressive=True): """Replace the standard socket object with gevent's cooperative sockets. ... _socket.socket = socket.socket # This line breaks multiprocessing.connection! ...，以及使用multiprocessing.connection而不是stdlib的gevent.socket模块有什么好处？也就是说，如果没有修补socket模块，会产生什么性能损失？

回溯

socket

重现错误的代码

（在ubuntu服务器11.10上，python2.7.3，安装了gevent，greenlet和grequests）

manager.py

Traceback (most recent call last):
  File "clientWithGeventMonkeyPatch.py", line 49, in <module>
    client = GetClient(host, port, authkey)
  File "clientWithGeventMonkeyPatch.py", line 39, in GetClient
    client.connect()
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 500, in connect
    conn = Client(self._address, authkey=self._authkey)
  File "/usr/lib/python2.7/multiprocessing/connection.py", line 175, in Client
    answer_challenge(c, authkey)
  File "/usr/lib/python2.7/multiprocessing/connection.py", line 414, in answer_challenge
    response = connection.recv_bytes(256)        # reject large message
IOError: [Errno 11] Resource temporarily unavailable

client.py

## manager.py
import multiprocessing
import multiprocessing.managers
import datetime


class LocalManager(multiprocessing.managers.SyncManager):
    def __init__(self, *args, **kwargs):
        multiprocessing.managers.SyncManager.__init__(self, *args, **kwargs)
        self.__type__ = 'LocalManager'

def GetManager(host, port, authkey):
    def getdatetime():
        return '{}'.format(datetime.datetime.now())

    LocalManager.register('getdatetime', callable = getdatetime)
    manager = LocalManager(address = (host, port), authkey = authkey)
    manager.start()

    return manager

if __name__ == '__main__':
    # define our manager connection parameters
    port = 55555
    host = 'localhost'
    authkey = 'auth1234'

    # start a manager
    man = GetManager(host, port, authkey)

    # wait for user input to shut down
    raw_input('return to shutdown')
    man.shutdown()

clientWithGeventMonkeyPatch.py

## client.py -- this one works
import time
import multiprocessing.managers

class RemoteClient(multiprocessing.managers.SyncManager):
    def __init__(self, *args, **kwargs):
        multiprocessing.managers.SyncManager.__init__(self, *args, **kwargs)
        self.__type__ = 'RemoteClient'

def GetClient(host, port, authkey):
    RemoteClient.register('getdatetime')
    client = RemoteClient(address = (host, port), authkey = authkey)
    client.connect()
    return client

if __name__ == '__main__':
    # define our client connection parameters
    port = 55555
    host = 'localhost'
    authkey = 'auth1234'

    # start a manager
    client = GetClient(host, port, authkey)
    print 'connected', client
    print 'client.getdatetime()', client.getdatetime()
    # wait a couple of seconds, then do it again
    time.sleep(2)
    print 'client.getdatetime()', client.getdatetime()

    # exit...

Answer 1

如果您没有修补套接字模块，gevent无法阻止网络操作的能力将无法使用，因此首先使用gevent的大部分好处将无法使用。

gevent和multiprocessing并不是真的可以很好地相互配合 - gevent主要假设您通过它进行网络连接，而不是绕过最高级别Python套接字接口（multiprocessing执行）。

为什么gevent.socket会破坏multiprocessing.connection的auth

回溯

重现错误的代码

1 个答案: