Question

使用多处理，我想传递一个可迭代的多个参数：

a）运行在n_core cpu上的函数 b）一次产生（或返回）n_core结果 c）按任何完成顺序

from multiprocessing import Pool 

def func(iterable, args):
    this, that, other = args[0], args[1], args[2]

    for s in iterable:
        return ' '.join([s, this, that, other])        

def main():
    iterable = ['abc', 'bcd', 'cde', 'def', 'efg', 'fgh', 'ghi', 'hij']
    args = ['this', 'that', 'other']
    n_core = 2

    p = Pool(n_core)
    for r in p.imap_unordered(func, iterable, args):
        print(r)

if __name__ == '__main__':
    main()

预期结果如下：

"abc this that other"
"bcd this that other"
"cde this that other"
"def this that other" 
"efg this that other" 
"fgh this that other"
"ghi this that other" 
"hij this that other"

使这项工作的正确方法是什么？

其次，concurrent.futures.ProcessPoolExecutor会更好地解决这个问题吗？

Answer 1

您可以创建new_iterable，将iterable中的值与args结合起来：

from multiprocessing import Pool

def func(args):
    iterable, this, that, other = args[0], args[1][0], args[1][1], args[1][2]
    return ' '.join([iterable, this, that, other])

def main():
    iterable = ['abc', 'bcd', 'cde', 'def', 'efg', 'fgh', 'ghi', 'hij']
    args = ['this', 'that', 'other']
    new_iterable = ([x, args] for x in iterable)
    n_core = 2

    p = Pool(n_core)
    for r in p.imap_unordered(func, new_iterable):
        print(r)

if __name__ == '__main__':
    main()

<强>输出

abc this that other
bcd this that other
cde this that other
def this that other
efg this that other
fgh this that other
ghi this that other
hij this that other

此解决方案使用generator expression创建一个新的可迭代，它将iterable中的条目与所需的args相结合。您也可以使用generator function执行相同的操作。

更新：我修改了func以生成您在评论中提到并添加到您的问题中的预期结果。

Answer 2

问题中的代码似乎有误。 <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script> <table class="flags-table" id="flags-table" responsive hover> <thead> <tr> <th></th> <th>Time In</th> <th>Time Out</th> <th>Type</th> <th>Category</th> </tr> </thead> <tbody> <tr data-time="08:00" data-stop-time="13:00" data-tag-name="test1" data-category="cat1" data-key="34098" class="low"> <td> <div className="red-box"></div> </td> <td>08:00</td> <td>13:00</td> <td>Tag 1</td> <td>Category 1</td> </tr> <tr data-time="09:00" data-stop-time="15:00" data-tag-name="test2" data-category="cat2" data-key="34096" class="low"> <td> <div className="red-box"></div> </td> <td>09:00</td> <td>15:00</td> <td>Tag 2</td> <td>Category 2</td> </tr> <tr data-time="03:00" data-stop-time="17:00" data-tag-name="test3" data-category="cat3" data-key="34095" class="low"> <td> <div className="red-box"></div> </td> <td>03:00</td> <td>17:00</td> <td>Tag 3</td> <td>Category 3</td> </tr> <tr data-time="06:00" data-stop-time="17:20" data-tag-name="test4" data-category="cat4" data-key="34094" class="low"> <td> <div className="red-box"></div> </td> <td>06:00</td> <td>17:20</td> <td>Tag 4</td> <td>Category 4</td> </tr> </tbody> </table>应该期望单个项目，而不是整个可迭代项目。

而不是：

func

您可以使用：

def func(iterable, args):
    this, that, other = args[0], args[1], args[2]

    for s in iterable:
        return ' '.join([s, this, that, other])

除此错误外，def func(item, args): this, that, other = args[0], args[1], args[2] return ' '.join([item, this, that, other])不接受多个参数。

此代码可以满足您的期望：

imap_unordered

具有可迭代和多个参数的Python多处理

2 个答案: