Question

mpi4py文档声称您可以通过MPIPoolExecutor传递** kwargs，但是我无法使其正常工作。我执行以下代码：

import time
import socket
import numpy as np
from mpi4py.futures import MPIPoolExecutor
from mpi4py import MPI


def print_val(x, kwargsstr):
    time.sleep(1)
    msg = "x value: " + str(x) + \
        ", socket: " + str(socket.gethostname()) + \
        ", rank: " + str(MPI.COMM_WORLD.Get_rank()) + \
        ", time: " + str(np.round(time.time(), 1)) + \
        ", kwargs string: " + kwargsstr
    print(msg)

    return x


def main():

    kwarg = 'test'
    kwargs = {'kwargsstr':kwarg}

    with MPIPoolExecutor() as executor:
        x_v = []
        # for res in executor.map(print_val,
        #                         range(9), 9*[kwarg]):
        for res in executor.map(print_val,
                                range(9), **kwargs):
            x_v += [res]
        print(x_v)


if __name__ == '__main__':

    """
    run on command line with 1 scheduler and 2 workers:
    $ mpiexec -n 1 -usize 3 -machinefile hostfile.txt python mpi4py_kwargs.py
    """

    main()

通过此命令：

$ mpiexec -n 1 -usize 3 -machinefile hostfile.txt python mpi4py_kwargs.py

并获得以下错误消息：

TypeError: print_val() missing 1 required positional argument: 'kwargsstr'

请注意，当切换main中注释掉的部分时，代码将按预期运行。

Answer 1

我不认为map支持kwargs。让我们看一下source code

map确实以kwargs作为签名：

def map(self, fn, *iterables, **kwargs):

这是文档中关于kwargs自变量的内容：

Keyword Args:
  unordered: If ``True``, yield results out-of-order, as completed.

除unordered之外的关键字参数是否将传递给可调用对象尚不明显。我们继续。该实现使用以下方式完成：

iterable = getattr(itertools, 'izip', zip)(*iterables)
return self.starmap(fn, iterable, **kwargs)

因此，kwargs被转发到starmap：

def starmap(self, fn, iterable, timeout=None, chunksize=1, **kwargs):

具有相同的文档

Keyword Args:
  unordered: If ``True``, yield results out-of-order, as completed.

实现如下：

unordered = kwargs.pop('unordered', False)
if chunksize < 1:
    raise ValueError("chunksize must be >= 1.")
if chunksize == 1:
    return _starmap_helper(self.submit, fn, iterable,
                           timeout, unordered)
else:
    return _starmap_chunks(self.submit, fn, iterable,
                           timeout, unordered, chunksize)

很显然，一旦检索到kwargs，就不再使用unordered。因此，map不会将kwargs视为可召唤的矮人。

正如我在评论中所指出的，对于此特定示例，您还可以将参数提供为非关键字参数，但也可以提供位置参数：

for res in executor.map(print_val, range(9), [kwarg] * 9):

作为map的第二个参数记录为：

iterables: Iterables yielding positional arguments to be passed to
           the callable.

对于python新手来说：

>>> kwarg = 'test'
>>> [kwarg] * 9
['test', 'test', 'test', 'test', 'test', 'test', 'test', 'test', 'test']

其中[kwarg] * 9是实现可迭代协议的列表。

** kwargs在mpi4py MPIPoolExecutor中不起作用

1 个答案: