Question

我在某处看到了如何使用多处理模块更快地处理大型数据集（比如说文本行），如：

... (form batch_set = nump batches [= lists of lines to process], batch_set
     is a list of lists of strings (batches))
nump = len(batch_set)
output = mp.Queue()
processes = [mp.Process(target=proc_lines, args=(i, output, batch_set[i])) for i in range(nump)]

for p in processes:
    p.start()
for p in processes:
    p.join()

results = sorted([output.get() for p in processes])
... (do something with the processed outputs, ex print them in order,
    given that each proc_lines function returns a couple (i, out_batch))

然而，当我使用少量行/批次运行代码时，它工作正常 [例如：'。/ code.py -x 4:10'为nump = 4，numb = 10（行/批））一定数量的行/批挂起[例如：'。/ code.py -x 4：4000']当我打断它我看到一个关于_wait_for_tstate_lock和系统的回溯提示线程库。似乎代码没有到达最后显示的代码以上......

我提供下面的代码，以防有人需要它来回答为什么会这样正在发生以及如何解决它。

#!/usr/bin/env python3

import sys
import multiprocessing as mp


def fabl(numb, nump):
    '''
    Form And Batch Lines: form nump[roc] groups of numb[atch] indexed lines
    '<idx> my line here' with indexes from 1 to (nump x numb).
    '''
    ret = []
    idx = 1
    for _ in range(nump):
        cb = []
        for _ in range(numb):
            cb.append('%07d my line here' % idx)
            idx += 1
        ret.append(cb)
    return ret


def proc_lines(i, output, rows_in):
    ret = []
    for row in rows_in:
        row = row[0:8] + 'some other stuff\n' # replacement for the post-idx part
        ret.append(row)

    output.put((i,ret))
    return


def mp_proc(batch_set):
    'given the batch, disperse it to the number of processes and ret the results'
    nump = len(batch_set)
    output = mp.Queue()
    processes = [mp.Process(target=proc_lines, args=(i, output, batch_set[i])) for i in range(nump)]

    for p in processes:
        p.start()
    for p in processes:
        p.join()

    print('waiting for procs to complete...')
    results = sorted([output.get() for p in processes])
    return results


def write_set(proc_batch_set, fout):
    'write p[rocessed]batch_set'
    for _, out_batch in proc_batch_set:
        for row in out_batch:
            fout.write(row)
    return


def main():
    args = sys.argv
    if len(args) < 2:
        print('''
    run with args: -x [ NumProc:BatchSize ]
        ( ex: '-x' | '-x 4:10' (default values) | '-x 4:4000' (hangs...) )
        ''')
        sys.exit(0)

    numb = 10  # suppose we need this number of lines/batch : BatchSize
    nump = 4   # number of processes to use.                : NumProcs
    if len(args) > 2 and ':' in args[2]: # use another np:bs
        nump, numb = map(int, args[2].split(':'))

    batch_set = fabl(numb, nump)  # proc-batch made in here: nump (groups) x numb (lines)
    proc_batch_set = mp_proc(batch_set)

    with open('out-min', 'wt') as fout:
        write_set(proc_batch_set, fout)

    return

if __name__ == '__main__':
    main()

Answer 1

TableCell具有一定的容量，如果在Queue运行时未将其清空，则可以填满。这不会阻止您的流程执行，但如果Process未完成，您将无法加入Process。

所以我只想修改put函数：

mp_proc

python3 multiprocessing.Process方法失败

1 个答案: