并行将多个泡菜保存到一个文件中

时间:2020-08-24 08:21:16

标签: python-3.x parallel-processing pickle joblib

我知道我可以将多个泡菜保存到一个文件中

import pickle
from joblib import Parallel, delayed

def g(val, f):
    ## do some stuff 
    t = "." + val
    pickle.dump(t, f)
    
def loadall(pick):
    with open(pick, 'rb') as f:
        while True:
            try:
                yield(pickle.load(f))
            except:
                break
    

fname = 'pickle.pickle'
with open(fname, 'wb') as f:
    [g(val, f) for val in list("ABCDEFG")] ## serial
    #Parallel(n_jobs=4)(delayed(g)(val, f) for val in list("ABCDEFG")) ## parallel

[print(r) for r in loadall(fname)]

有什么办法可以并行执行此操作吗?切换到并行循环会引发异常: TypeError:无法腌制'_io.BufferedWriter'对象

我知道我可以保存不同的泡菜并在处理结束时将它们加入,但是写入同一文件更加干净。

0 个答案:

没有答案