Question

我正在尝试将一些python 2代码移植到python 3.我现在面临着多进程模块的一些困难。更确切地说，下面的代码片段在Python 2.7.11中按预期工作（它使用subprocess.Popen生成一个新进程并重定向stdout）。

import time
import pickle
import subprocess
from multiprocessing import Process, Queue, Manager

manager = Manager()

def execute(popen):
    stdout_lines = iter(popen.stdout.readline, "")
    for stdout_line in stdout_lines:
        yield stdout_line
    popen.stdout.close()
    return_code = popen.wait()

q = manager.Queue()
popen = subprocess.Popen(["python", "-u", "helloworld.py"],
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        universal_newlines=True,
        bufsize=1)

q.put(popen)

try:
    for l in execute(popen):
        try:
            print(l)
        except socket.error as e:
            break
        except AttributeError as e:
            break
        except IOError as e:
            break
except:
    print("Finished.")

q.get().kill()

文件helloworld.py包含每2秒打印helloworld的脚本。

import time

while True:
    print("helloworld")
    time.sleep(2)

但是，当使用python 3.5.3运行时，它会失败，因为它无法序列化

`TextIOWrapper` object, which seems to be related to the redirected stream.
Traceback (most recent call last):
  File "./test_subprocess.py", line 22, in <module>
    q.put(popen)
  File "<string>", line 2, in put
  File "/Users/scheibler/anaconda/envs/python3/lib/python3.5/multiprocessing/managers.py", line 716, in _callmethod
    conn.send((self._id, methodname, args, kwds))
  File "/Users/scheibler/anaconda/envs/python3/lib/python3.5/multiprocessing/connection.py", line 206, in send
    self._send_bytes(ForkingPickler.dumps(obj))
  File "/Users/scheibler/anaconda/envs/python3/lib/python3.5/multiprocessing/reduction.py", line 50, in dumps
    cls(buf, protocol).dump(obj)
TypeError: cannot serialize '_io.TextIOWrapper' object

一些研究产生了这种相关的question。然而，它似乎是Windows特定的，作者提到他的代码在linux下工作。我还应该指定我在OS X下使用Anaconda python运行代码。

multiprocessing.Queue无法排队python3

0 个答案: