Question

我有一个代码，可以读取非常大的文本文件并处理Pools中的每一行。

对于 elif ，我需要将整个过程休眠120秒，换句话说，我希望所有创建的其他Pool都暂停。但是在120秒后，所有的Pool应该恢复工作。

代码的功能与此类似：

from multiprocessing import Pool
import sys

sys.tracebacklimit = 0

def req(line):

    if "@" not in line:
        # (some function for processing here)
        return line
    elif "somestring" in line:
        #HERE I NEED TO SLEEP ALL POOLS
    else:
        # (some function for processing)
        return line


if __name__ == "__main__":
    pool = Pool(20)
    with open("list.txt") as source_file:
        # chunk the work into batches of 20 lines at a time
        pool.map(req, source_file, 35)

Answer 1

如@abarnert所述，您应按以下方式使用Event对象：

from multiprocessing import Pool
import sys
from threading import Event, Timer

sys.tracebacklimit = 0

# Setup clojure environment
def reqgen():
    ev_stop = Event()

    def req(line):

        # Wait at the start
        if ev_stop.is_set():
            ev_stop.wait()

        if "somestring" in line:
            #HERE I NEED TO SLEEP ALL POOLS

            # Clear the internal flag, make all workers await
            ev_stop.clear()

            # An alarm to reset the internal flag,
            # which will make all workers get back to work
            Timer(120, lambda: ev_stop.set()).start()

            # Regular work
            return req(line)

        else:
            # (some function for processing)
            return line

    return req

if __name__ == "__main__":
    pool = Pool(20)
    with open("list.txt") as source_file:
        # chunk the work into batches of 20 lines at a time
        pool.map(reqgen(), source_file, 35)

在if语句检查中休眠线程，Python池

1 个答案: