Question

   def multi_run_wrapper(self, args):
       return self.parrallel(*args)


    try:
        pool = multiprocessing.Pool(maxtasksperchild=1)
        chunksize, extra = divmod(len(dictionary), cpu_count() * 4)
        if extra:
            chunksize += 1
        results = pool.imap_unordered(self.multi_run_wrapper, dictionary.iteritems(), chunksize)
    except Exception as ex:
        print "Error in pool", str(ex)
    finally:
        pool.close()
        pool.join()

我正在使用multiprocessing来加快执行速度。基本函数根据函数的输入在不同的网址上执行request.get和request.post。这在我的mac上正常运行但是在运行时linux vm通过jenkins它挂起。我已经尝试了所有选项，例如map,imap,map_async。我正在使用pandas所以我也尝试检查cpu_affinity但是我的正确设置为16 case.I也试过没有maxtasksperchild。我现在已经用尽了所有选项。还有其他我可以试试的吗？

P.S：当jenkins进程挂起时...当我通过jenkins终止时，它仍然保留在ps - ef | grep python我运行的相同脚本中。

运行gdb后，我可以看到3个进程在等待。似乎有一个僵局!!!!

挂起时的线程信息

info threads
  Id   Target Id         Frame 
  4    Thread 0x7f4172227700 (LWP 6623) 0x00007f41e1a595e3 in select ()
at ../sysdeps/unix/syscall-template.S:82
  3    Thread 0x7f4171826700 (LWP 6624) 0x00007f41e23a56fd in write () at ../sysdeps/unix/syscall-template.S:82
  2    Thread 0x7f41d1186700 (LWP 6625) 0x00007f41e23a575d in read () at ../sysdeps/unix/syscall-template.S:82
* 1    Thread 0x7f41e27c5700 (LWP 6196) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86

Thread 4 (Thread 0x7f4172227700 (LWP 6623)):
Traceback (most recent call first):
  (frame information optimized out)
  File "/usr/local/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
(frame information optimized out)

Thread 3 (Thread 0x7f4171826700 (LWP 6624)):
Traceback (most recent call first):
(frame information optimized out)
   File "/usr/local/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
   File "/usr/local/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
(frame information optimized out)

Thread 2 (Thread 0x7f41d1186700 (LWP 6625)):
Traceback (most recent call first):
(frame information optimized out)
File "/usr/local/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/local/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
(frame information optimized out)

Thread 1 (Thread 0x7f41e27c5700 (LWP 6196)):
Traceback (most recent call first):
Waiting for the GIL
(frame information optimized out)
(frame information optimized out)
File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 457, in join
self._worker_handler.join()
 File "/home/jenkins/fsroot/workspace/testvivekpds/utils/r_p/RRFP.py", line 942, in u_r_t_p
pool.join()
(frame information optimized out)
(frame information optimized out)

Jenkins VM下的多处理挂起

0 个答案: