Question

我已经实现了这样的解析器，

import multiprocessing
import time

def foo(i):
    try:
        # some codes
    except Exception, e:
        print e

def worker(i):
    foo(i)
    time.sleep(i)
    return i


if __name__ == "__main__":
    pool = multiprocessing.Pool(processes=4)
    result = pool.map_async(worker, range(15))
    while not result.ready():
        print("num left: {}".format(result._number_left))
        time.sleep(1)
    real_result = result.get()
    pool.close()
    pool.join()

我的解析器实际上完成了所有进程，但结果不可用，即它仍在while循环内并打印num left : 2。我怎么阻止这个？我不希望real_result变量的值。

我正在运行Ubuntu 14.04，python 2.7

我的代码的相应部分看起来像，

    async_args = ((date, kw_dict) for date in dates)
    pool = Pool(processes=4)
    no_rec = []

    def check_for_exit(msg):
        print msg
        if last_date in msg:
            print 'Terminating the pool'
            pool.terminate()
    try:
        result = pool.map_async(parse_date_range, async_args)
        while not result.ready():
            print("num left: {}".format(result._number_left))
            sleep(1)

        real_result = result.get(5)

        passed_dates = []

        for x, y in real_result:
            passed_dates.append(x)
            if y:
                no_rec.append(y[0])

        # if last_date in passed_dates:
        #     print 'Terminating the pool'
        #     pool.terminate()

        pool.close()
    except:

        print 'Pool error'
        pool.terminate()
        print traceback.format_exc()
    finally:
        pool.join()

Answer 1

我敢打赌你有错误parse_date_range，这导致工作进程终止而不产生任何结果或py异常。由于一个令人讨厌的错误，可能由C模块/ lib调用libc的exit。
此代码重现您观察到的无限循环：

import sys
import multiprocessing
import time

def parse_date_range(i):
    if i == 5:
        sys.exit(1) # or raise SystemExit; 
                    # other exceptions are handled by the pool
    time.sleep(i/19.)
    return i


if __name__ == "__main__":
    pool = multiprocessing.Pool(4)
    result = pool.map_async(parse_date_range, range(15))
    while not result.ready():
        print("num left: {}".format(result._number_left))
        time.sleep(1)
    real_result = result.get()
    pool.close()
    pool.join()

希望这会有所帮助。

完成所有任务后，Python multiprocessing.pool无法停止

1 个答案: