Question

我正在尝试并行化一些动态LCA（就像10 ......而我应该运行超过一千......）我得到了这个

Can't expand MemType 1: jcol 19852
Can't expand MemType 0: jcol 17930
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/giuseppec/miniconda3/envs/bw2/lib/python3.4/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/giuseppec/miniconda3/envs/bw2/lib/python3.4/multiprocessing/pool.py", line 47, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "multiprocess_both.py", line 129, in dyn_lca
    timeline=dynamic_lca.calculate()
  File "/home/giuseppec/miniconda3/envs/bw2/lib/python3.4/site-packages/bw2temporalis/dynamic_lca.py", line 54, in calculate
    **self.gt_kwargs
  File "/home/giuseppec/miniconda3/envs/bw2/lib/python3.4/site-packages/bw2calc/graph_traversal.py", line 44, in calculate
    lca, supply, score = self.build_lca(demand, method)
  File "/home/giuseppec/miniconda3/envs/bw2/lib/python3.4/site-packages/bw2calc/graph_traversal.py", line 107, in build_lca
    lca.decompose_technosphere()
  File "/home/giuseppec/miniconda3/envs/bw2/lib/python3.4/site-packages/bw2calc/lca.py", line 216, in decompose_technosphere
    self.solver = factorized(self.technosphere_matrix.tocsc())
  File "/home/giuseppec/miniconda3/envs/bw2/lib/python3.4/site-packages/scipy/sparse/linalg/dsolve/linsolve.py", line 377, in factorized
    return splu(A).solve
  File "/home/giuseppec/miniconda3/envs/bw2/lib/python3.4/site-packages/scipy/sparse/linalg/dsolve/linsolve.py", line 257, in splu
    ilu=False, options=_options)
MemoryError
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "multiprocess_both.py", line 181, in <module>
    dyn_multi=[x for x in pool_dyn_multi.get()] 
  File "/home/giuseppec/miniconda3/envs/bw2/lib/python3.4/multiprocessing/pool.py", line 608, in get
    raise self._value
MemoryError

似乎scipy在矩阵反转期间耗尽了内存（我当然不是最有见识的，但我搜索了一下）我只是尝试安装scikit-umfpack以查看是否有变化....它正在运行但仍然看起来非常缓慢且不太有希望而且只是返回此警告：

Warning: (almost) singular matrix! (estimated cond. number: 3.33e+14)

实际上对如何处理这个问题一无所知......任何人都有？

P.S：我发布在这里，因为它看起来不是bw2的错误，如果是这样就会打开一个问题

Answer 1

你的游泳池大小是多少？可能最简单的尝试是减少并行运行的作业数量，但将cpu的数量设置为可用数量的一半。

某处可能存在内存泄漏，并且在作业完成后内存未被释放。您可以在constructing a MP Pool时设置maxtasksperchild=1来回避这一点。

稀疏矩阵求解器在并行动态LCA中耗尽内存

1 个答案: