我正在使用其多处理模块编写Python程序。该程序调用许多工作函数,每个函数产生一个随机数。 我需要在其中一个工人生成一个大于0.7 的数字后终止程序。
以下是我的程序,其中" 如何执行此操作"部分尚未填写。任何的想法?感谢。
import time
import numpy as np
import multiprocessing as mp
import time
import sys
def f(i):
np.random.seed(int(time.time()+i))
time.sleep(3)
res=np.random.rand()
print "From i = ",i, " res = ",res
if res>0.7:
print "find it"
# terminate ???? Question: How to do this???
if __name__=='__main__':
num_workers=mp.cpu_count()
pool=mp.Pool(num_workers)
for i in range(num_workers):
p=mp.Process(target=f,args=(i,))
p.start()
答案 0 :(得分:17)
没有任何过程可以阻止另一种蛮力os.kill()
- 就像大锤一样。不要去那里。
要做到这一点,您需要重新设计基本方法:主要流程和工作流程需要相互沟通。
我充实了它,但到目前为止的例子太只是为了让它变得有用。例如,正如所写的那样,对num_workers
的{{1}}次呼叫不会超过rand()
,因此没有理由相信它们中的任何一个必须是> 0.7。
一旦worker函数生成一个循环,它就会变得更加明显。例如,工作人员可以检查是否在循环顶部设置了mp.Event
,如果是,则退出。当需要工人停止时,主要流程会设置Event
。
当工作人员找到值>时,可以设置不同的mp.Event
。 0.7。主进程将等待Event
,然后将"时间设置为停止" Event
让工人看到,然后通常循环.join()
- 让工人彻底关闭。
这里充实了一个便携,干净的解决方案,假设工人将继续前进,直到至少有人找到一个值> 0.7。请注意,我从中删除了numpy
,因为它与此代码无关。这里的代码在任何支持multiprocessing
的平台上的任何股票Python都可以正常工作:
import random
from time import sleep
def worker(i, quit, foundit):
print "%d started" % i
while not quit.is_set():
x = random.random()
if x > 0.7:
print '%d found %g' % (i, x)
foundit.set()
break
sleep(0.1)
print "%d is done" % i
if __name__ == "__main__":
import multiprocessing as mp
quit = mp.Event()
foundit = mp.Event()
for i in range(mp.cpu_count()):
p = mp.Process(target=worker, args=(i, quit, foundit))
p.start()
foundit.wait()
quit.set()
一些示例输出:
0 started
1 started
2 started
2 found 0.922803
2 is done
3 started
3 is done
4 started
4 is done
5 started
5 is done
6 started
6 is done
7 started
7 is done
0 is done
1 is done
一切都干净利落:没有追溯,没有异常终止,没有留下僵尸进程......干净如哨声。
正如@noxdafox指出的那样,有一种Pool.terminate()
方法可以跨平台尽最大努力杀死工作进程,无论他们在做什么(例如,在Windows上调用平台TerminateProcess()
)。我不推荐它用于生产代码,因为突然终止进程可能会使各种共享资源处于不一致状态,或者让它们泄漏。 multiprocessing
文档中有各种警告,您应该添加操作系统文档。
不过,这可能是权宜之计!这是使用这种方法的完整程序。请注意,我将截止值提高到了0.95,这使得这更有可能比使用eyeblink更长的时间:
import random
from time import sleep
def worker(i):
print "%d started" % i
while True:
x = random.random()
print '%d found %g' % (i, x)
if x > 0.95:
return x # triggers callback
sleep(0.5)
# callback running only in __main__
def quit(arg):
print "quitting with %g" % arg
# note: p is visible because it's global in __main__
p.terminate() # kill all pool workers
if __name__ == "__main__":
import multiprocessing as mp
ncpu = mp.cpu_count()
p = mp.Pool(ncpu)
for i in range(ncpu):
p.apply_async(worker, args=(i,), callback=quit)
p.close()
p.join()
一些示例输出:
$ python mptest.py
0 started
0 found 0.391351
1 started
1 found 0.767374
2 started
2 found 0.110969
3 started
3 found 0.611442
4 started
4 found 0.790782
5 started
5 found 0.554611
6 started
6 found 0.0483844
7 started
7 found 0.862496
0 found 0.27175
1 found 0.0398836
2 found 0.884015
3 found 0.988702
quitting with 0.988702
4 found 0.909178
5 found 0.336805
6 found 0.961192
7 found 0.912875
$ [the program ended]
答案 1 :(得分:1)
正如其他用户提到的那样,您需要流程相互通信才能让他们终止他们的同行。虽然您可以使用os.kill来终止对等进程,但发送终止信号更为优雅。
我使用的解决方案非常简单: 1.找出主进程的进程ID(pid),它产生所有其他工作进程。此连接信息可从操作系统获得,该操作系统可跟踪从哪个父进程生成哪个子进程。 2.当其中一个工作进程达到最终条件时,它使用父进程ID查找主进程的所有子进程(包括其自身),然后遍历列表并发出信号结束(确保它不是信令本身) 下面的代码包含工作解决方案。
import time
import numpy as np
import multiprocessing as mp
import time
import sys
import os
import psutil
import signal
pid_array = []
def f(i):
np.random.seed(int(time.time()+i))
time.sleep(3)
res=np.random.rand()
current_process = os.getpid()
print "From i = ",i, " res = ",res, " with process ID (pid) = ", current_process
if res>0.7:
print "find it"
# solution: use the parent child connection between processes
parent = psutil.Process(main_process)
children = parent.children(recursive=True)
for process in children:
if not (process.pid == current_process):
print "Process: ",current_process, " killed process: ", process.pid
process.send_signal(signal.SIGTERM)
if __name__=='__main__':
num_workers=mp.cpu_count()
pool=mp.Pool(num_workers)
main_process = os.getpid()
print "Main process: ", main_process
for i in range(num_workers):
p=mp.Process(target=f,args=(i,))
p.start()
输出清楚地了解发生了什么:
Main process: 30249
From i = 0 res = 0.224609517693 with process ID (pid) = 30259
From i = 1 res = 0.470935062176 with process ID (pid) = 30260
From i = 2 res = 0.493680214732 with process ID (pid) = 30261
From i = 3 res = 0.342349294134 with process ID (pid) = 30262
From i = 4 res = 0.149124648092 with process ID (pid) = 30263
From i = 5 res = 0.0134122107375 with process ID (pid) = 30264
From i = 6 res = 0.719062852901 with process ID (pid) = 30265
find it
From i = 7 res = 0.663682945388 with process ID (pid) = 30266
Process: 30265 killed process: 30259
Process: 30265 killed process: 30260
Process: 30265 killed process: 30261
Process: 30265 killed process: 30262
Process: 30265 killed process: 30263
Process: 30265 killed process: 30264
Process: 30265 killed process: 30266
答案 2 :(得分:1)
通过使用multiprocessing.Pool
提供的回调函数,有一种更清洁和pythonic的方式来做你想做的事情。
您可以查看this question以查看实施示例。
答案 3 :(得分:-3)
您只需从sys
导入exit()
即可终止您的程序
import sys
sys.exit()