Question

我遇到了BrokenPipeError：[Errno 32]使用Pathos多进程程序包时管道错误。基本上，每个过程都运行有模型训练过程。就个人而言，我认为这似乎是一个记忆问题。但是，任何建议将不胜感激！

  Process ForkPoolWorker-9:
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 125, in worker
    put((job, i, result))
  File "/home/ubuntu/.local/lib/python3.5/site-packages/sklearn/externals/joblib/pool.py", line 386, in put
    return send(obj)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/sklearn/externals/joblib/pool.py", line 372, in send
    self._writer.send_bytes(buffer.getvalue())
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

在处理上述异常期间，发生了另一个异常：

Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 130, in worker
    put((job, i, (False, wrapped)))
  File "/home/ubuntu/.local/lib/python3.5/site-packages/sklearn/externals/joblib/pool.py", line 386, in put
    return send(obj)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/sklearn/externals/joblib/pool.py", line 372, in send
    self._writer.send_bytes(buffer.getvalue())
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

Answer 1

@Isiah Meadows，像这样。建模代码运行良好，并且其他输入数据流正在运行相同的过程。

from pathos.pools import ProcessPool
pool = ProcessPool(5)
try:
    segment_lists = pool.map(segmentModelMaster, list(args_iteration))
    segment_grid_list, segment_evl_list = zip(*segment_lists)
except Exception:
    logger.error('Multi-process individual task failed, aborting!')
    pool.close()
    pool.terminate()

BrokenPipeError：[Errno 32]管道损坏

1 个答案: