Question

首先为我犯的任何错误道歉。我是StackOverflow的新手，我仍然习惯它。

背景：我尝试通过多处理训练多个keras模型。那些Keras模型存储在三个列表中（每个列表当前有50个模型）。我使用的功能有三个模型作为输入（每个列表中的一个）以及一些训练数据和ID。输出应该是评估分数和给定的ID。

我的第一次尝试是从标准python库导入多处理。

from multiprocessing import Pool

事实证明，这个多处理应用程序不起作用，因为pickle作为序列化工具。所以我下载了Pathos和Dill来处理这个问题。我的代码是这样的：

    from pathos.multiprocessing import ProcessingPool as Pool
    p = Pool()
    from pathos.helpers import freeze_support
    import dill

    if __name__ == "__main__":
        freeze_support()

        result = p.amap(training.train_critic, models_actor_t, models_critic, models_critic_t, batch_states_list, batch_actions_list, batch_rewards_list, batch_new_states_list , batch_terminals_list, sub_epochs_list, id_list)

        print(result)

如果我运行此代码，一切正常，我得到了这个输出。

<multiprocess.pool.MapResult object at 0x000001D42CEC09E8>

如果我添加方法result.get（），麻烦就会开始。

   if __name__ == "__main__":
        freeze_support()

        result = p.amap(training.train_critic, models_actor_t, models_critic, models_critic_t, batch_states_list, batch_actions_list, batch_rewards_list, batch_new_states_list , batch_terminals_list, sub_epochs_list, id_list)

        print(result)
        print(result.get())

出局是这样的：

Traceback (most recent call last):


File "C:/Users/Lennart/PycharmProjects/demonstrator_RLAlgorithm/GeneticAlgorithm/GAoperations.py", line 192, in <module>
    print(result.get())
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\site-packages\multiprocess\pool.py", line 608, in get
    raise self._value
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\site-packages\multiprocess\pool.py", line 385, in _handle_tasks
    put(task)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\site-packages\multiprocess\connection.py", line 209, in send
    self._send_bytes(ForkingPickler.dumps(obj))
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\site-packages\multiprocess\reduction.py", line 53, in dumps
    cls(buf, protocol).dump(obj)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 408, in dump
    self.save(obj)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 744, in save_tuple
    save(element)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 729, in save_tuple
    save(element)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 729, in save_tuple
    save(element)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 729, in save_tuple
    save(element)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 744, in save_tuple
    save(element)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 520, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 627, in save_reduce
    save(state)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\site-packages\dill\dill.py", line 871, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 814, in save_dict
    self._batch_setitems(obj.items())
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 840, in _batch_setitems
    save(v)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 774, in save_list
    self._batch_appends(obj)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 801, in _batch_appends
    save(tmp[0])
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 520, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 627, in save_reduce
    save(state)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self

此后的最后一部分重演：

 File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\site-packages\dill\dill.py", line 871, in save_module_dict
    StockPickler.save_dict(pickler, obj)

所以我的问题是：

有什么根本错误吗？
Keras模型存储在列表中。因此，通过将它们插入到函数中，模型的权重仍然是持久的，对吧？

再次，如果我做了任何愚蠢的新手，我很抱歉。

谢谢，

伦纳特

无法通过池进行多处理获得结果

0 个答案: