Question

我想一次启动多个10个工作岗位，然后等待他们完成工作，然后在后台重启另外10个工作，重复此工作，直到完成所有100个工作。

这是调用shell脚本的python代码

from subprocess import call

# other code here.

# This variable is basically # of jobs/batch.
windowsize = 10

# Here is how I call the shell command. I have 100 jobs in total that I want as 10 batches with 10 jobs/batch.
for i in range (0..100) :
   numjobs = i + windowsize

   # Start 10 jobs in parallel at a time 
   for j in range (i..numjobs) :
       call (["./myscript.sh", "/usr/share/file1.txt", ""/usr/share/file2.txt"],   shell=True)

   # Hoping that to wait until the 10 jobs that were recently started in background finish.
   call(["wait],shell=True)

在我的shell脚本中，我有这个

#!/bin/sh

# I start the job in background. Each job takes few minutes to finish.

shell command $1 $2 &
...

不幸的是，所有100个作业都已启动，而不是10个批次，每批10个作业。

Answer 1

没有（直接）方式等待孙子进程。改为在wait脚本的末尾添加myscript.sh。

要限制并发运行的子进程数，可以使用线程池：

#!/usr/bin/env python
import logging
from multiprocessing.pool import ThreadPool
from subprocess import call

windowsize = 10
cmd = ["./myscript.sh", "/usr/share/file1.txt", "/usr/share/file2.txt"]

def run(i):
    return i, call(cmd)

logging.basicConfig(format="%(asctime)-15s %(message)s", datefmt="%F %T",
                    level=logging.INFO)
pool = ThreadPool(windowsize)
for i, rc in pool.imap_unordered(run, range(100)):
    logging.info('%s-th command returns %s', i, rc)

注意：shell=True已被删除。

在后台启动并等待多个作业

1 个答案: