我使用的是Python 3.5,并希望使用多处理模块来运行一个函数来解析一些文件,每个文件都转到不同的CPU核心;我向函数发送了很多参数,所以我使用了kwargs。
以下代码使用并发期货(线程),但我想将其转换为多处理等价物(并行期货 - ProcessPoolExecutor对我来说太重了,反正给我一些酸洗错误。)
from concurrent import futures
with futures.ThreadPoolExecutor(max_workers=4) as executor:
# A dictionary which will contain a list the future info in the key, and the filename in the value
jobs = {}
# Loop through the files, and run the parse function for each file, sending the file-name to it, along with the kwargs of parser_variables.
# The results of the functions can come back in any order.
for this_file in files_list:
job = executor.submit(parse_log_file.parse, this_file, **parser_variables)
jobs[job] = this_file
# Get the completed jobs whenever they are done
for job in futures.as_completed(jobs):
debug.checkpointer("Multi-threaded Parsing File finishing")
# Send the result of the file the job is based on (jobs[job]) and the job (job.result)
result_content = job.result()
this_file = jobs[job]
结果重新出现的顺序并不重要。
当我尝试使用pool.apply
,pool.apply_async
和pool.map
时(尽管阅读this question,我对这些差异的了解并不多)我得到了与kwarg相关的内容错误:
TypeError: apply() got an unexpected keyword argument 'variable_list'
如何将上述内容转换为多处理等效?