Question

我正在尝试使用p_tqdm通过p_map()库运行一些代码以并行化一些代码。我遇到了与dill相关的错误，但我无法弄清。

Traceback (most recent call last):
  File "C:\Users\uid\AppData\Local\Programs\Python\Python38\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Users\uid\AppData\Local\Programs\Python\Python38\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\uid\AppData\Local\Programs\Python\Python38\lib\site-packages\multiprocess\pool.py", line 576, in _handle_results
    task = get()
  File "C:\Users\uid\AppData\Local\Programs\Python\Python38\lib\site-packages\multiprocess\connection.py", line 254, in recv
    return _ForkingPickler.loads(buf.getbuffer())
  File "C:\Users\uid\AppData\Local\Programs\Python\Python38\lib\site-packages\dill\_dill.py", line 275, in loads
    return load(file, ignore, **kwds)
  File "C:\Users\uid\AppData\Local\Programs\Python\Python38\lib\site-packages\dill\_dill.py", line 270, in load
    return Unpickler(file, ignore=ignore, **kwds).load()
  File "C:\Users\uid\AppData\Local\Programs\Python\Python38\lib\site-packages\dill\_dill.py", line 473, in load
    obj = StockUnpickler.load(self)
TypeError: __init__() takes 1 positional argument but 2 were given

我的代码的结构为：

import pickle
from p_tqdm import p_map

def my_func(data_fp):
    data = pickle.load(open(data_fp, 'rb'))

    # Do basic stuff to the data here.

    return True

class MyClass():
    def process_data(self):
        # Do prep stuff here....get a list of filepaths we will need to load.

        ret = p_map(my_func, data_fp_list, num_cpus=0.75)

        return True

（{process_data()是从另一个文件的if __name__ == '__main__': multiprocessing.freeze_support()块中调用的）

在同时运行Python 3.8.2的Windows 10计算机和RHEL7计算机上，我都遇到了这种故障。我的pip list给出：

dill            0.3.2
filelock        3.0.12
future          0.18.2
idna            2.9
joblib          0.14.1
kiwisolver      1.2.0
matplotlib      3.2.1
multiprocess    0.70.10
numpy           1.18.3
p-tqdm          1.3.3
packaging       20.4
pandas          1.0.3
pathos          0.2.6
Pillow          7.1.1
pip             20.1.1
pox             0.2.8
ppft            1.6.6.2
pyparsing       2.4.7
pytesseract     0.3.4
python-dateutil 2.8.1
pytz            2020.1
regex           2020.5.7
requests        2.23.0
sacremoses      0.0.43
scikit-learn    0.23.1
scipy           1.4.1
sentencepiece   0.1.90
setuptools      41.2.0
six             1.14.0
threadpoolctl   2.1.0
tokenizers      0.8.1rc1
torch           1.5.0+cpu
torchvision     0.6.0+cpu
tqdm            4.45.0
transformers    3.0.2
urllib3         1.25.9
Wand            0.5.9

我已经看到有关pathos的类似问题，因为它与dill/pickle有关，但我不太理解。我和this question的处境相同吗？

更新：

内置multiprocessing库map函数也会发生这种情况，但是如果您使用线程而不是进程，则不会发生这种情况。 concurrent.futures.ThreadPoolExecutor.map()的魅力十足。我不知道这是否表明我正在腌制的数据有问题。

带有莳萝的Pathos p_tqdm多进程错误

0 个答案: