多重处理:无法腌制_thread.RLock对象

时间:2020-07-27 23:46:58

标签: python multiprocessing python-3.6 pickle dill

在Flask应用程序中,我有一个在for循环中调用的函数,我想在继续执行以下过程直到完成for循环之前,不执行该过程。

这是非常基本的场景的片段:

def get_most_similar_names(self):
    original_name = 'ABC ELEMENTARY EDUCATION'
    name_list = ['ABC ELEMENTARY SCHOOL', 'ABC KINDERGARDEN', 'ABC MIDDLE SCHOOL', 'ABC SOLUTIONS']
    vector1 = self.text_to_vector(original_name)
    manager = multiprocessing.Manager()
    return_dict = manager.dict()
    jobs = []
    for name in name_list:
        print('vector1: ', vector1, "type: ", type(vector1))
        print('name: ', name, " type: ", type(name))
        print("return_dict: ", return_dict, "type: ", type(return_dict))
        # For testing
        a = dill.pickles(vector1)
        b = dill.pickles(name)
        c = dill.pickles(return_dict)

        p = multiprocessing.Process(target=self.get_score, args=(vector1, name, return_dict))
        jobs.append(p)
        p.start()

    for proc in jobs:
        proc.join()

    print(return_dict.values())


def get_score(self, vec1, n, return_d):
    vec2 = self.text_to_vector(n)
    print("in get_cosine function")
    intersection = set(vec1.keys()) & set(vec2.keys())
    numerator = sum([vec1[x] * vec2[x] for x in intersection])

    sum1 = sum([vec1[x] ** 2 for x in list(vec1.keys())])
    sum2 = sum([vec2[x] ** 2 for x in list(vec2.keys())])

    denominator = math.sqrt(sum1) * math.sqrt(sum2)

    if not denominator:
        cosine = 0.0
    else:
        cosine = float(numerator) / denominator

    return_d[n] = cosine

    print("return_dict:", return_d)

    return return_d

目的是执行get_score函数,该函数接收vector1,name和一个共享变量以将结果存储在字典中。但是,执行start()时出现以下错误:

File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle _thread.RLock objects

我认为这些参数不是可腌制的,所以我对其进行了测试,但是它们 是可腌制的。我已经知道记录器会产生此错误,我在其他模块中找到了current_app.logger执行。

您知道其他因素会影响start()的执行吗?

0 个答案:

没有答案