Question

我尝试从存档（tar）读取txt文件并使用函数处理文件。我有很多文件，并决定使用多处理库来加速计算。但我收到错误“无法序列化'ExFileObject'对象”。下面是我的代码示例。如何序列化'ExFileObject'对象或如何使用多处理lib加速我的计算？我正在使用Python 3.4

import tarfile
from multiprocessing import Process, Manager

def start():
    result = {}
    file_path = '...somepath'
    tar = tarfile.open(file_path, 'r')
    for file in tar:
        extr_file = tar.extractfile(file)
        p = Process(target=straight_calc, args=(extr_file, result)) #thows error
        p.start()
        p.join()
    print(result)

def straight_calc(file, result):
    content = file.readlines()
    for line in content:
        pass

if __name__ == '__main__':
    start()

Answer 1

您的示例代码适用于我。我只会添加NoneType项检查，因为NoneType没有readlines属性，而tar.extractfile(file)可能会返回NoneType值。

import tarfile
from multiprocessing import Process, Manager

def start():
    result = {}
    file_path = '...somepath'
    tar = tarfile.open(file_path, 'r')
    for file in tar:
        extr_file = tar.extractfile(file)
        if extr_file is not None:
            p = Process(target=straight_calc, args=(extr_file, result))
            p.start()
            p.join()
    print(result)

def straight_calc(file, result):
    content = file.readlines()
    for line in content:
        pass

if __name__ == '__main__':
    start()

您可以尝试使用picklestring = pickle.dumps(extr_file)来搜索extr_file。在执行Process之前pickling extr_file应该抛出一个更有用的异常。

Python 3.x：Multiprocess无法序列化'ExFileObject'对象

1 个答案: