Question

我正在使用Python 2.7 multiprocessing.Queue，其父进程是队列的“生产者”（即任务“调度程序”）和作为队列使用者的子进程（即“工作者” “过程”。我需要子进程非“守护进程”，这样，如果在父进程中发生异常，它将不会中断子进程对现有队列项的处理。但是，我发现，如果父进程发生异常，则会发生死锁，因为父进程尝试加入子进程，但是一旦队列清空，子进程将永远等待更多队列项。另一方面，如果我使子进程“守护进程”，那么子进程在父进程从异常终止时终止，这会中断子进程对现有队列项的处理。

演示示例（使用非守护进程子进程）：

import multiprocessing, time

def childProcessFunction(sharedQueue):
    # Consumer loop
    while True:
        print "Child process: Consuming item.  We will deadlock here if the queue is empty and the parent process has terminated!"
        consumedItem=sharedQueue.get()

        print "Child process: Taking some time to process the consumed item..."
        time.sleep(5)
        print consumedItem # Do something with (print) the consumed item.
        time.sleep(5)
        print "Child process: Done processing the consumed item; re-looping to get another item..."

def parentProcessMain():
    sharedQueue=multiprocessing.Queue()

    childProcess=multiprocessing.Process(target=childProcessFunction, args=(sharedQueue,))

    # Don't want a daemon process because we don't want to interrupt the child's 
    # processing of existing items when the parent process terminates.
    childProcess.daemon=False

    print "Parent process: Starting child process..."
    childProcess.start()

    failAfter=3 # Simulate a failure after producing 3 items.

    numDispatched=0 # Number of items put onto the queue.

    # Producer loop
    while True:
        time.sleep(3)

        if failAfter==0:
            raise Exception("Parent process encountered a failure of some kind and is exiting uncleanly!  Now the parent process will try to join the child process, and the child process will deadlock once the queue is empty!")

        producedItem="This is item number: %d"%numDispatched
        print "Parent process: Enqueueing item number %d..."%numDispatched
        sharedQueue.put(producedItem)
        numDispatched+=1
        failAfter-=1
        time.sleep(3)

if __name__=="__main__":
    parentProcessMain()

是否有一种方法可以允许子进程在父进程获得异常（非守护进程行为）之后完成处理任何正在进行的处理以及队列中剩余的任何其他内容，但是阻止子进程从此后永远等待队列？有没有办法让队列“意识到”生产者已终止并在get()上抛出异常而不是无限期阻塞的事实？

请注意，如果队列恰好为空并且子进程在父进程异常时在get()调用中等待，那么在这种情况下我需要get()终止（可能是例外）。

Answer 1

在您知道子进程完成之前，您能否保留异常？也许通过某种形式的同步，如互斥锁。

当生成器终止异常时，Python多处理队列死锁

1 个答案: