Question

我想使用多处理将我的函数（f1）应用于数字数组（cdr_test）。我的代码：

cdr_test = [x for x in range(0, 100000)]

def f1(el):
    a = Counter() #make new vector for each cdr
    for k,v in d3.items():
        if el in v:
            a = a + Counter(itertools.product([el], v))  
    return a

if __name__ == '__main__':
    pool = mp.Pool(20)
    results = pool.map(f1, cdr_test)

    pool.close()
    pool.join() 

    out = open('out.txt', 'w')
    for result in results:
        for k,v in result.items():
            out.write('\t'.join(map(str,k))+"\t"+str(v)+"\n")
    out.close()  
    pool.close()

我无法分配内存＆＃39;。如果我使用较小长度的数组（100），那么一切正常。

堆栈跟踪：

OSError                                   Traceback (most recent call last)
<ipython-input-3-b8dc4a3d12b3> in <module>()
     9 
     10 if __name__ == '__main__':
---> 11     pool = mp.Pool(1000)
     12     results = pool.map(f1, cdr_test)
     13     #new section

    /home/fedorovaad/anaconda3/lib/python3.5/multiprocessing/context.py in   Pool(self, processes, initializer, initargs, maxtasksperchild)
116         from .pool import Pool
117         return Pool(processes, initializer, initargs, maxtasksperchild,
--> 118                     context=self.get_context())
    119 
    120     def RawValue(self, typecode_or_type, *args):

/home/fedorovaad/anaconda3/lib/python3.5/multiprocessing/pool.py in      __init__(self, processes, initializer, initargs, maxtasksperchild, context)
    166         self._processes = processes
    167         self._pool = []
--> 168         self._repopulate_pool()
    169 
    170         self._worker_handler = threading.Thread(

/home/fedorovaad/anaconda3/lib/python3.5/multiprocessing/pool.py in  _repopulate_pool(self)
    231             w.name = w.name.replace('Process', 'PoolWorker')
    232             w.daemon = True
--> 233             w.start()
    234             util.debug('added worker')
    235 

/home/fedorovaad/anaconda3/lib/python3.5/multiprocessing/process.py in start(self)
    103                'daemonic processes are not allowed to have  children'
    104         _cleanup()
--> 105         self._popen = self._Popen(self)
    106         self._sentinel = self._popen.sentinel
    107         _children.add(self)

/home/fedorovaad/anaconda3/lib/python3.5/multiprocessing/context.py in _Popen(process_obj)
    265         def _Popen(process_obj):
    266             from .popen_fork import Popen
--> 267             return Popen(process_obj)
    268 
    269     class SpawnProcess(process.BaseProcess):

/home/fedorovaad/anaconda3/lib/python3.5/multiprocessing/popen_fork.py in __init__(self, process_obj)
     18         sys.stderr.flush()
     19         self.returncode = None
---> 20         self._launch(process_obj)
     21 
     22     def duplicate_for_child(self, fd):

/home/fedorovaad/anaconda3/lib/python3.5/multiprocessing/popen_fork.py in _launch(self, process_obj)
     65         code = 1
     66         parent_r, child_w = os.pipe()
---> 67         self.pid = os.fork()
     68         if self.pid == 0:
     69             try:

OSError: [Errno 12] Cannot allocate memory

有办法解决这个问题吗？

Answer 1

您显示的代码与错误中的代码不同。

---> 11     pool = mp.Pool(1000)

你试图产生过多的进程，操作系统会在它们全部分配之前耗尽内存。

您不需要这么多流程来继续工作，只需使用multiprocessing.cpu_count()来了解您的平台拥有多少CPU并产生该大小的池。

无法在多处理python中分配内存

1 个答案: