Question

我有一个实例化对象的sconstruct脚本。该对象在内部调用一个方法，而不是运行多处理模块。示例如下所示

调用函数之前，此对象取消选择文件并将输入传递给多处理模块。

def run_scons(self,inpfile,outfile):

        # Unpickle input parameter
        fid=open(inpfile,'rb')
        input_data=pkls.load(fid)
        my_results=[]
        #run solver in loop
        for my_data in input_data:
            work_ers=len(my_data)
            pool = Pool(processes=work_ers)
            a_result=pool.map_async(my_solver, my_data)
            pool.close()
            pool.join()
            my_results.append(a_result.get())
        fid.close()

        fid_out=open(outfile,'wb+')
        pkls.dump(rot_full_results,fid_out)

通过scons执行相同的功能时出现以下错误。

pool = Pool(processes=work_ers)
  File "C:\Python27\lib\multiprocessing\__init__.py", line 232, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild)
  File "C:\Python27\lib\multiprocessing\pool.py", line 138, in __init__
    self._setup_queues()
  File "C:\Python27\lib\multiprocessing\pool.py", line 232, in _setup_queues
    from .queues import SimpleQueue
  File "C:\Python27\lib\multiprocessing\queues.py", line 48, in <module>
    from multiprocessing.synchronize import Lock, BoundedSemaphore, Semaphore, Condition
  File "C:\Python27\lib\multiprocessing\synchronize.py", line 48, in <module>
    from multiprocessing.forking import assert_spawning, Popen
  File "C:\Python27\lib\multiprocessing\forking.py", line 60, in <module>
    class ForkingPickler(Pickler):
  File "C:\Python27\lib\multiprocessing\forking.py", line 61, in ForkingPickler
    dispatch = Pickler.dispatch.copy()
AttributeError: 'builtin_function_or_method' object has no attribute 'dispatch'
scons: building terminated because of errors.

在阅读了这个错误之后，我发现SCONS有一个黑客周围它将pickle模块重命名为cPickle，而多处理模块正在寻找cPickle并且一切都会失败。有没有办法解决？

Answer 1

我正在使用多处理来并行处理一些“作业”文件。我在我的SConstruct文件中做了类似的事情：

if GetOption("run_jobs"):

    my_jobs = # some code that produces a list of job objects

    ecode = build_support.mp_run_jobs(my_jobs)

    Exit(ecode)

但后来我遇到了上面的例外：

AttributeError: 'builtin_function_or_method' object has no attribute 'dispatch'

在使用'SCONS_HORRIBLE_REGRESSION_TEST_HACK'的Linux上工作：

$ SCONS_HORRIBLE_REGRESSION_TEST_HACK=1 scons --run-jobs

但我不想在Linux，Windows和Mac等上导出该符号。

我确实找到了解决方法，删除了pickle模块并重新导入它们：

if GetOption("run_jobs"):

    # Workaround SCons.compat module renaming

    import imp

    del sys.modules['pickle']
    del sys.modules['cPickle']

    sys.modules['pickle'] = imp.load_module('pickle', *imp.find_module('pickle'))
    sys.modules['cPickle'] = imp.load_module('cPickle', *imp.find_module('cPickle'))

    import pickle
    import cPickle

    print "(pickle == cPickle) = ", (pickle == cPickle)

    my_jobs = # some code that produces a list of job objects

    ecode = build_support.mp_run_jobs(my_jobs)

    Exit(ecode)

现在，我的多工作处理按预期工作！

Scons属性错误：'builtin_function_or_method'对象没有属性'dispatch'

1 个答案: