Question

我在下面写了一个调用函数（调用外部程序）的函数。我把它写成仅限于所提供的核心数，并且在分配更多内容之前等待所有当前任务完成。

我确信有一种更好的方法 - 我如何观察整理过程然后分配另一个？我已经看到了启动服务器和创建进程守护进程的事情，但是文档不是很好（或者至少我无法弄清楚）。

def parallel_lastz(target, chromosomes, sequence, gene, cores):
    """
    Calls lastz from <target> to <sequence> over each chromosome in <chromosomes>
    Using <cores>, waiting for each thread to finish before continuing.
    Probably could be made better but at least its faster than 1 at a time
    """
    fasta = ">{}\n{}\n".format(gene, sequence)
    alignments = list()
    tmp = list()
    pool = Pool(processes=cores)
    for i in xrange(len(chromosomes)):
        if (i + 1) % p != 0:
            chrom_path = "{}/{}.fasta".format(target, chromosomes[i])
            tmp.append(pool.apply_async(call_lastz, (chrom_path, fasta)))
        else:
            for x in tmp:
                alignments.append(x.get())
            tmp = list()
    return alignments

Answer 1

这个怎么样：

def worker( chromosome ) :
  chrom_path = "{}/{}.fasta".format(target, chromosome )
  return call_lastz(chrom_path, fasta)

if __name__ == '__main__' :
  pool = multiprocessing.Pool(processes=cores)
  alignments = pool.map( worker, chromosomes )
  pool.close()
  pool.join()

使用python multiprocess - 在完成后立即在进程上启动一个新函数（比核心更多的作业）

1 个答案: