设置子进程cpu亲和性,使用mpiexec运行 - >作业调度程序无法杀死作业

时间:2016-06-09 12:41:46

标签: multiprocessing cluster-computing openmpi job-scheduling psutil

[[问题]]

我使用psutil库来设置子进程的cpu亲和性,并使用mpirun和job scheduler运行。

从作业调度程序中删除作业后,我将ssh插入节点。

当我使用ps aux检查时,python主进程和11个python子进程仍然继续运行并更新日志。

只有mpirun进程被杀死。

在设置python子进程的cpu亲和性时,psutil库不会更改pid。

不使用mpirun运行它,作业调度程序可以毫无问题地终止python子进程。

[[问题]]

当作业调度程序删除作业时,如何使进程实际被杀死?

感谢。

[[测试代码]]

import logging
import psutil
import time
import multiprocessing as mp
def main():
    logging.basicConfig(format='%(asctime)s %(message)s', level=logging.INFO)
    # set_process_affinity(0)
    num_cpu = mp.cpu_count()
    logging.info('call worker')

    # Starts num_cpu - 1 subprocesses
    # For a node with 12 cpu, this makes 11 processes.

    mp_pool = mp.Pool(processes = num_cpu - 1)
    result_list = [ mp_pool.apply_async(worker,(i,)) for i in range(1,num_cpu)]
    mp_pool.close()
    mp_pool.join()

def set_process_affinity(cpu_id):
    psutil_proc = psutil.Process()
    logging.info("cpu_id #%d :: proc_info_before %s" % (cpu_id, psutil_proc))
    psutil_proc.cpu_affinity([cpu_id])
    psutil_proc = psutil.Process()
    logging.info("cpu_id #%d :: proc_info_after %s" % (cpu_id, psutil_proc))

def worker(worker_id):
    cpu_id = worker_id
    set_process_affinity(cpu_id)
    logging.basicConfig(format='%(asctime)s %(message)s', level=logging.INFO)
    for cycle in range(10000):
        logging.info("worker #%d :: cycle %d" % (worker_id, cycle))
        waste_time = []
        for i in xrange(1000000):
            waste_time += [i]
        #time.sleep(10)

if __name__ == '__main__':
    main()

[[作业脚本中的命令]]

使用mpirun(openmpi-1.8.1):

mpirun -np 1 --map-by node python2.7 -u ps_test.py &> ps_test.log

没有mpirun:

python2.7 -u ps_test.py &> ps_test.log

[[用ssh检查节点,然后运行ps aux | grep“rxu”]]

与mpirun:

=在使用任务调度程序(使用mpirun)=

查杀作业之前
rxu       3686  0.3  0.0 105856  4396 ?        Sl   11:15   0:00 mpirun -np 1 --map-by node python2.7 -u ps_test.py
rxu       3688  0.6  0.1 172408 11660 ?        Sl   11:15   0:00 python2.7 -u ps_test.py
rxu       3689 96.0  0.4 206692 40656 ?        R    11:15   0:15 python2.7 -u ps_test.py
rxu       3690 96.0  0.4 206692 40660 ?        R    11:15   0:15 python2.7 -u ps_test.py
rxu       3691 96.0  0.4 206692 40676 ?        R    11:15   0:15 python2.7 -u ps_test.py
rxu       3692 96.0  0.4 206692 40680 ?        R    11:15   0:15 python2.7 -u ps_test.py
rxu       3693 96.0  0.4 206696 40688 ?        R    11:15   0:15 python2.7 -u ps_test.py
rxu       3694  102  0.4 206696 40684 ?        R    11:15   0:15 python2.7 -u ps_test.py
rxu       3695  102  0.4 203328 40684 ?        R    11:15   0:15 python2.7 -u ps_test.py
rxu       3696  102  0.4 203328 40668 ?        R    11:15   0:15 python2.7 -u ps_test.py
rxu       3697  102  0.4 203328 40668 ?        R    11:15   0:15 python2.7 -u ps_test.py
rxu       3698  101  0.4 203328 40668 ?        R    11:15   0:15 python2.7 -u ps_test.py
rxu       3699  102  0.4 203332 40672 ?        R    11:15   0:15 python2.7 -u ps_test.py
... some processes from pts/1 including ssh into the node

=使用任务调度程序(使用mpirun)=

查杀作业后
The mpirun process get killed.
The python main process (the one with 0.0 cpu used) lives
The 11 python subprocess lives (none got killed).
The machine has 12 cpu.

rxu       3688  0.0  0.1 172408 11660 ?        Sl   11:15   0:00 python2.7 -u ps_test.py
rxu       3689 99.6  0.4 206692 40708 ?        R    11:15   3:34 python2.7 -u ps_test.py
rxu       3690 99.6  0.4 206692 40732 ?        R    11:15   3:34 python2.7 -u ps_test.py
rxu       3691 99.6  0.4 206692 40724 ?        R    11:15   3:34 python2.7 -u ps_test.py
rxu       3692 99.6  0.4 206692 40728 ?        R    11:15   3:34 python2.7 -u ps_test.py
rxu       3693 99.6  0.4 206696 40736 ?        R    11:15   3:34 python2.7 -u ps_test.py
rxu       3694  100  0.4 206696 40732 ?        R    11:15   3:34 python2.7 -u ps_test.py
rxu       3695  100  0.4 203328 40732 ?        R    11:15   3:34 python2.7 -u ps_test.py
rxu       3696 99.9  0.4 203328 40720 ?        R    11:15   3:33 python2.7 -u ps_test.py
rxu       3697  100  0.4 203328 40720 ?        R    11:15   3:34 python2.7 -u ps_test.py
rxu       3698 99.9  0.4 203328 40720 ?        R    11:15   3:33 python2.7 -u ps_test.py
rxu       3699  100  0.4 203332 40724 ?        R    11:15   3:34 python2.7 -u ps_test.py

=我得到的日志(使用mpirun)=

(The pid didn't change upon setting cpu affinity of the subprocess)

2016-06-09 11:15:45,280 call worker
2016-06-09 11:15:45,333 cpu_id #1 :: proc_info_before psutil.Process(pid=3689, name='python2.7')
2016-06-09 11:15:45,334 cpu_id #1 :: proc_info_after psutil.Process(pid=3689, name='python2.7')
2016-06-09 11:15:45,335 worker #1 :: cycle 0
2016-06-09 11:15:45,335 cpu_id #2 :: proc_info_before psutil.Process(pid=3690, name='python2.7')
2016-06-09 11:15:45,336 cpu_id #2 :: proc_info_after psutil.Process(pid=3690, name='python2.7')

没有mpirun:

=在使用任务调度程序(没有mpirun)=

查杀作业之前
main process is the one with 0.3 %cpu used
11 works are those with 101 %cpu used.
There are 12 cpu on the machine.

rxu       3399  0.3  0.1 237940 11660 ?        Sl   11:06   0:00 python2.7 -u ps_test.py
rxu       3400  101  0.4 206688 40648 ?        R    11:06   0:35 python2.7 -u ps_test.py
rxu       3401  101  0.4 206688 40652 ?        R    11:06   0:35 python2.7 -u ps_test.py
rxu       3402  101  0.4 206688 40672 ?        R    11:06   0:35 python2.7 -u ps_test.py
rxu       3403  101  0.4 206688 40676 ?        R    11:06   0:35 python2.7 -u ps_test.py
rxu       3404  101  0.4 206692 40684 ?        R    11:06   0:35 python2.7 -u ps_test.py
rxu       3405  101  0.4 206692 40680 ?        R    11:06   0:35 python2.7 -u ps_test.py
rxu       3406  101  0.4 203324 40680 ?        R    11:06   0:35 python2.7 -u ps_test.py
rxu       3407  101  0.4 203324 40664 ?        R    11:06   0:35 python2.7 -u ps_test.py
rxu       3408  101  0.4 203324 40664 ?        R    11:06   0:35 python2.7 -u ps_test.py
rxu       3409  101  0.4 203324 40664 ?        R    11:06   0:35 python2.7 -u ps_test.py
rxu       3410  101  0.4 203328 40668 ?        R    11:06   0:35 python2.7 -u ps_test.py

... some processes from pts/1 including ssh into the node

=使用任务计划程序(不使用mpirun)=

终止作业后
nothng. no python processes
... except some processes from pts/1 including ssh into the node

0 个答案:

没有答案