我想创建一个程序来生成多个子进程,然后读取它们的输出。
每个子进程将使用 curl 从服务器读取数据并每隔几秒钟刷新一次该数据。为简单起见,可以将其替换为 subproc1 的简单 cat <filename1>
,subproc2 的 cat <filename2>
等。
然后(父)程序进程从子进程 stdin 读取,保持顺序,进行一些解析并打印到 stdout。为简单起见,假设它回显以“subproc[1-N]”为前缀的输入
我已经尝试使用 subprocess.Popen
、Queue
和守护线程进行读写。但我无法让它按预期工作。
这是一个使用 select 的例子。
#!/usr/local/bin/python3
import subprocess, shlex
import sys, os
import select
import pdb
cmds=[ "ls -l", "ls -a", "./print_nums.sh"]
procs = [ subprocess.Popen(shlex.split(cmd), stdin=subprocess.PIPE, stdout=subprocess.PIPE) for cmd in cmds ]
streams = [p.stdout for p in procs]
streams2Cmds = {k:v for (k,v) in zip(streams, cmds)}
def handler(line, stream):
print('{}:\n{}'.format(streams2Cmds[stream], line.decode('utf-8')))
while True:
rstreams,_,_ = select.select(streams, [], [])
for stream in rstreams:
print('handle input from {}'.format(streams2Cmds[stream]))
line = stream.readline()
handler(line, stream)
if all(p.poll is not None for p in procs):
break
for stream in streams:
handler(stream.read(), stream)
它工作正常,只是它没有为每一行输入添加前缀。 输出将类似于:
handle input from ls -l
ls -l:
total 120
handle input from ls -a
ls -a:
.
ls -l:
-rwxr-xr-x 1 i500695 staff 779 Jan 24 14:31 popen2.orig.py
-rwxr-xr-x 1 i500695 staff 1000 Jan 24 14:17 popen2.orig2.py
-rwxr-xr-x 1 i500695 staff 990 Jan 24 14:13 popen2.py
-rwxr-xr-x 1 i500695 staff 1553 Mar 1 2020 popen2_no_select.py
-rw-r--r-- 1 i500695 staff 2264 Mar 1 2020 popen2_no_select.py2
-rw-r--r-- 1 i500695 staff 1519 Jan 24 14:25 popenv2.py
-rw-r--r-- 1 i500695 staff 775 Jan 24 14:23 popenv3.py
-rwxr-xr-x 1 i500695 staff 53 Jan 24 13:25 print_nums.sh
-rw-r--r-- 1 i500695 staff 227 Jan 13 2020 proc_a.py
prw-r--r-- 1 i500695 staff 0 Jan 24 12:10 proc_a_input
-rw-r--r-- 1 i500695 staff 227 Jan 13 2020 proc_b.py
prw-r--r-- 1 i500695 staff 0 Jan 24 12:10 proc_b_input
-rw-r--r-- 1 i500695 staff 1075 Jan 24 12:16 readTwoStdings.py
-rwxr-xr-x 1 i500695 staff 503 Jan 13 2020 simple_popen.py
-rw-r--r-- 1 i500695 staff 3013 Jan 24 12:12 two_subprocesses.py
-rw-r--r-- 1 i500695 staff 712 Jan 13 2020 two_subprocesses2.py
-rw-r--r-- 1 i500695 staff 1461 Jan 13 2020 two_subprocesses_with_output.py
ls -a:
..
popen2.orig.py
popen2.orig2.py
popen2.py
popen2_no_select.py
popen2_no_select.py2
popenv2.py
popenv3.py
print_nums.sh
proc_a.py
proc_a_input
proc_b.py
proc_b_input
readTwoStdings.py
simple_popen.py
two_subprocesses.py
two_subprocesses2.py
two_subprocesses_with_output.py
./print_nums.sh:
1
2
3
4
5