无法导入多线程Python模块。

Question

这是我试图作为模块导入的多线程代码。当我将它作为独立文件运行时，它工作正常。它只是打印出一个数字列表。我改变的是main（）被注释掉了。我认为问题可能是导入的代码在读取器文件已将其称为模块之后将其自身称为模块。

threadtest.py

from multiprocessing import Process, Value, Array
import time
import sys

def f(n, a):
    n.value = 3.1415927
    for i in range(len(a)):
        a[i] = -a[i]

def is_prime(x, top):
    for j in range(2,int(top**.5 + 1)):
        if x%j == 0:
            return False
    return True

def f1(top, c):
    print('test f1')
    for i in range(2,top):
        if is_prime(i,top):
            c.value = c.value + 1

def f3(c1,c2):
    for k in range(0,20):
        time.sleep(1)  # 1 second
        sys.stdout.write(str(c1.value) + '|' + str(c2.value) + '\n')  
        sys.stdout.flush()  

def main():

count1 = Value('d', 0)
count2 = Value('d', 0)
#arr = Array('i', range(10))
p1 = Process(target=f1, args=(1000000, count1))
p2 = Process(target=f1, args=(1000000, count2))
p3 = Process(target=f3, args=(count1, count2))

p1.start()
p2.start()
p3.start()

p1.join()
print('p1.join()')
p2.join()
print('p2.join()')
p3.join()
print('p3.join()')

print(count1.value)
print(count2.value)

if __name__ == '__main__':
    print('grovetest is being run as main')
    #main()
else:
    print('grovetest is being run as module')
    main()

这是导入多线程模块并尝试读取输出的代码。

readertest.py

import threadtest

def main(fname):

    try:

        p = subprocess.Popen(fname, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
        print('success')
        return p.communicate()  # this gets you pipe values

    except Exception as e:

        return 'error'+str(e)

    else:

        return "else"

if __name__ == '__main__':
    main(threadtest)

以下是运行readertest.py

时产生的错误

grovetest is being run as module
grovetest is being run as module
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 106, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 115, in _main
    prepare(preparation_data)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 226, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 278, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 240, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Joseph\documents\python projects\multitest.py", line 1, in <module>
    import grovetest
  File "C:\Users\Joseph\documents\python projects\grovetest.py", line 56, in <module>
    main()
  File "C:\Users\Joseph\documents\python projects\grovetest.py", line 37, in main
    p1.start()
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 212, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 313, in _Popen
    return Popen(process_obj)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\popen_spawn_win32.py", line 34, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 144, in get_preparation_data
    _check_not_importing_main()
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 137, in _check_not_importing_main
    is not going to be frozen to produce an executable.''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 106, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 115, in _main
    prepare(preparation_data)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 226, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 278, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 240, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Joseph\documents\python projects\multitest.py", line 1, in <module>
    import grovetest
  File "C:\Users\Joseph\documents\python projects\grovetest.py", line 56, in <module>
    main()
  File "C:\Users\Joseph\documents\python projects\grovetest.py", line 37, in main
    p1.start()
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 212, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 313, in _Popen
    return Popen(process_obj)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\popen_spawn_win32.py", line 34, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 144, in get_preparation_data
    _check_not_importing_main()
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 137, in _check_not_importing_main
    is not going to be frozen to produce an executable.''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
grovetest is being run as module
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 106, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 115, in _main
    prepare(preparation_data)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 226, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 278, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 240, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Joseph\documents\python projects\multitest.py", line 1, in <module>
    import grovetest
  File "C:\Users\Joseph\documents\python projects\grovetest.py", line 56, in <module>
    main()
  File "C:\Users\Joseph\documents\python projects\grovetest.py", line 37, in main
    p1.start()
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 212, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 313, in _Popen
    return Popen(process_obj)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\popen_spawn_win32.py", line 34, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 144, in get_preparation_data
    _check_not_importing_main()
  File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 137, in _check_not_importing_main
    is not going to be frozen to produce an executable.''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
grovetest is being run as module
p1.join()
p2.join()
p3.join()
0.0
0.0

我对此很陌生。感谢您的反馈。

Answer 1

您违反了一个名为here的重要Windows特定约束：

  

确保新的Python解释器可以安全地导入主模块，而不会导致意外的副作用（例如启动新进程）。

我刚才引用的段落中的一个缺陷是它说'＃34;主要模块＆＃34;何时应该说＆＃34; 每个模块＆＃34;，但是大多数人都不会尝试从导入的模块运行复杂的代码，因此不要遇到这个问题。在您的情况下，会发生readertest导入grovetest，它调用多处理代码，使用特殊的Windows多进程引导系统在grovetest上触发一个新的Python解释器，该系统将启动多处理代码，它检测到它在特殊引导程序下被触发并中止：

(start)
readertest:
    import grovetest

 => grovetest
    import multiprocessing
     => multiprocessing: define some stuff

 => grovetest
    if __name__ == '__main__': # it's not equal
        ... # so we skip this
    else:
        print('grovetest is being run as module') # prints
        main() # calls main
    ...
    p1 = Process(target=f1, args=(1000000, count1)) # calls multiprocessing module

      => new process
         import grovetest # multiprocessing does this using its magic bootstrap
         ...
         if __name__ == '__main__': # it's not
             ...
         else:
             print(...)
             ...
             p1 = Process(...)

此时内部Process代码检测到约束违规并引发第一个错误。这完全终止了内部流程，我们又回到了p2 = Process(...)的外部流程，它再次启动了相同的错误，依此类推。

要解决此问题，当main作为模块导入时，一定不能在grovetest中调用grovetest。而是从grovetest.main致电readertest 以后。这将允许特殊引导魔法（在multiprocessing模块中，在您调用multiprocessing.Process时运行）以在新​​进程中导入grovetest。一旦这个新进程导入grovetest，它将等待来自其父进程（另一个Python解释器，其中你称为multiprocessing.Process）的指令来自readertest代码调用grovetest.main）。这些指令将使用提供的参数调用函数f1（来自target=f1）。