使用kwargs进行多处理;转换使用并发期货

时间:2016-01-02 19:47:48

标签: python python-3.x concurrency multiprocessing

我使用的是Python 3.5,并希望使用多处理模块来运行一个函数来解析一些文件,每个文件都转到不同的CPU核心;我向函数发送了很多参数,所以我使用了kwargs。

以下代码使用并发期货(线程),但我想将其转换为多处理等价物(并行期货 - ProcessPoolExecutor对我来说太重了,反正给我一些酸洗错误。)

from concurrent import futures
with futures.ThreadPoolExecutor(max_workers=4) as executor:
    # A dictionary which will contain a list the future info in the key, and the filename in the value
    jobs = {}

    # Loop through the files, and run the parse function for each file, sending the file-name to it, along with the kwargs of parser_variables.
    # The results of the functions can come back in any order.
    for this_file in files_list:
        job = executor.submit(parse_log_file.parse, this_file, **parser_variables)
        jobs[job] = this_file

    # Get the completed jobs whenever they are done
    for job in futures.as_completed(jobs):
        debug.checkpointer("Multi-threaded Parsing File finishing")

        # Send the result of the file the job is based on (jobs[job]) and the job (job.result)
        result_content = job.result()
        this_file = jobs[job]

结果重新出现的顺序并不重要。

当我尝试使用pool.applypool.apply_asyncpool.map时(尽管阅读this question,我对这些差异的了解并不多)我得到了与kwarg相关的内容错误:

TypeError: apply() got an unexpected keyword argument 'variable_list'

如何将上述内容转换为多处理等效?

0 个答案:

没有答案