我有一组异步作业(大约100个),我希望每个作业使用subprocess.popen
批量运行五个。我的计划是:
在python中有没有已知的模式?
答案 0 :(得分:2)
在Python 2中,我使用了multiprocessing.Pool
和subprocess
的组合。但是这确实会以池的进程形式产生额外的开销。
所以在Python 3中我使用concurrent.futures.ThreadPoolExecutor
而不是multiprocessing.pool
;
下面的代码片段显示了如何使用ThreadPoolExecutor
;
import concurrent.futures as cf
import logging
import os
errmsg = 'conversion of track {} failed, return code {}'
okmsg = 'finished track {}, "{}"'
num = len(data['tracks'])
with cf.ThreadPoolExecutor(max_workers=os.cpu_count()) as tp:
fl = [tp.submit(runflac, t, data) for t in range(num)]
for fut in cf.as_completed(fl):
idx, rv = fut.result()
if rv == 0:
logging.info(okmsg.format(idx+1, data['tracks'][idx]))
else:
logging.error(errmsg.format(idx+1, rv))
runflac
功能使用subprocess
来呼叫flac(1)
转换音乐文件:
import subprocess
def runflac(idx, data):
"""Use the flac(1) program to convert a music file to FLAC format.
Arguments:
idx: track index (starts from 0)
data: album data dictionary
Returns:
A tuple containing the track index and return value of flac.
"""
num = idx + 1
ifn = 'track{:02d}.cdda.wav'.format(num)
args = ['flac', '--best', '--totally-silent',
'-TARTIST=' + data['artist'], '-TALBUM=' + data['title'],
'-TTITLE=' + data['tracks'][idx],
'-TDATE={}'.format(data['year']),
'-TGENRE={}'.format(data['genre']),
'-TTRACKNUM={:02d}'.format(num), '-o',
'track{:02d}.flac'.format(num), ifn]
rv = subprocess.call(args, stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL)
return (idx, rv)
<强>更新强>
在Python 2.7中,还有一种技术略微复杂,但避免了使用多处理池的开销。
基本形式是:
starter = functools.partial(startencoder, crf=args.crf, preset=args.preset)
procs = []
maxprocs = cpu_count()
for ifile in args.files:
while len(procs) == maxprocs:
manageprocs(procs)
procs.append(starter(ifile))
while len(procs) > 0:
manageprocs(procs)
(使用functools.partial
是一种为函数设置默认参数的方法。它与原理无关。)startencoder
函数基本上是subprocess.Popen
的包装,但它返回除Popen
实例之外的一些额外信息;
def startencoder(fname, crf, preset):
"""
Use ffmpeg to convert a video file to H.264/AAC streams in an MP4
container.
Arguments:
fname: Name of the file to convert.
crf: Constant rate factor. See ffmpeg docs.
preset: Encoding preset. See ffmpeg docs.
Returns:
A 3-tuple of a Process, input path and output path.
"""
basename, ext = os.path.splitext(fname)
known = ['.mp4', '.avi', '.wmv', '.flv', '.mpg', '.mpeg', '.mov', '.ogv',
'.mkv', '.webm']
if ext.lower() not in known:
ls = "File {} has unknown extension, ignoring it.".format(fname)
logging.warning(ls)
return (None, fname, None)
ofn = basename + '.mp4'
args = ['ffmpeg', '-i', fname, '-c:v', 'libx264', '-crf', str(crf),
'-preset', preset, '-flags', '+aic+mv4', '-c:a', 'libfaac',
'-sn', '-y', ofn]
try:
p = subprocess.Popen(args, stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL)
logging.info("Conversion of {} to {} started.".format(fname, ofn))
except:
logging.error("Starting conversion of {} failed.".format(fname))
return (p, fname, ofn)
重要的是manageprocs
功能:
def manageprocs(proclist):
"""
Check a list of subprocesses tuples for processes that have ended and
remove them from the list.
Arguments:
proclist: a list of (process, input filename, output filename)
tuples.
"""
nr = '# of conversions running: {}\r'.format(len(proclist))
logging.info(nr)
sys.stdout.flush()
for p in proclist:
pr, ifn, ofn = p
if pr is None:
proclist.remove(p)
elif pr.poll() is not None:
logging.info('Conversion of {} to {} finished.'.format(ifn, ofn))
proclist.remove(p)
sleep(0.5)