Question

我有这个愚蠢的代码来解释我在工作中遇到的行为：

from multiprocessing import Pool
from concurrent.futures import ProcessPoolExecutor
from struct import pack
from time import time


def packer(integer):

    return pack('i', integer)


if __name__=='__main__':

    pool1 = Pool()
    pool2 = ProcessPoolExecutor()

    nums = list(range(10**4))

    start = time()
    res1 = pool1.map(packer, nums)
    print (f'total mp pool: {time() - start}')

    start = time()
    res2 = pool2.map(packer, nums)
    print (f'total futures pool: {time() - start}')

    pool1.close()

我得到了（Python 3.8.1）：

total mp pool: 0.1600649356842041
total futures pool: 0.9206013679504395

在工作中，我将代码从mp.Pool()修改为concurrent.futures，以允许在进程和线程之间移动。

然后我发现concurrent.futures的异常传播很时髦。回到mp.Pool()，发现性能有所下降。

我知道concurrent.futures.ProcessPoolExecutor应该是更高级别的API，它比mp.Pool()快吗？

我看到here ProcessPoolExecutor.map很简单：

super().map(partial(_process_chunk, fn),
                              _get_chunks(*iterables, chunksize=chunksize),
                              timeout=timeout)

其中super是_base.Executor：

def map(self, fn, *iterables, timeout=None, chunksize=1):
    """Returns an iterator equivalent to map(fn, iter).
    Args:
        fn: A callable that will take as many arguments as there are
            passed iterables.
        timeout: The maximum number of seconds to wait. If None, then there
            is no limit on the wait time.
        chunksize: The size of the chunks the iterable will be broken into
            before being passed to a child process. This argument is only
            used by ProcessPoolExecutor; it is ignored by
            ThreadPoolExecutor.
    Returns:
        An iterator equivalent to: map(func, *iterables) but the calls may
        be evaluated out-of-order.
    Raises:
        TimeoutError: If the entire result iterator could not be generated
            before the given timeout.
        Exception: If fn(*args) raises for any values.
    """
    if timeout is not None:
        end_time = timeout + time.monotonic()

    fs = [self.submit(fn, *args) for args in zip(*iterables)]

    # Yield must be hidden in closure so that the futures are submitted
    # before the first iterator value is required.
    def result_iterator():
        try:
            # reverse to keep finishing order
            fs.reverse()
            while fs:
                # Careful not to keep a reference to the popped future
                if timeout is None:
                    yield fs.pop().result()
                else:
                    yield fs.pop().result(end_time - time.monotonic())
        finally:
            for future in fs:
                future.cancel()
    return result_iterator()

这是我有点迷路的地方。

mp.Pool和ProcessPoolExecutor跌入了另一个兔子洞吗？是否可以通过正确的参数手动调用ProcessPoolExecutor / mp / Pool来以某种方式从map获得“好东西”？

为什么mp.Pool（）。map（）比ProcessPoolExecutor（）。map（）

0 个答案: