python多处理：AttributeError：无法获得属性＆＃34; abc＆＃34;

Question

我正在尝试运行一个简单的命令，使用多处理按名称猜测性别。这段代码适用于以前的机器，所以也许我的设置与它有关。

下面是我的多处理代码：

import sys
import gender_guesser.detector as gender
import multiprocessing
import time

d = gender.Detector()

def guess_gender (name):
    n = name.title() # make first letter upper case and the rest lower case 
    g = d.get_gender(n) # guess gender
    return g

ls = ['john','joe','amamda','derick','peter','ashley','john','joe','amamda','derick','peter','ashley']

t=time.time()

results=[]
def callBack(x):
    results.append(x)

pool = multiprocessing.Pool(processes=multiprocessing.cpu_count()-1, maxtasksperchild=1)

for n in ls:
    print (n)
    pool.apply_async(guess_gender,args=[n],callback=callBack)

pool.close()
pool.join()

results = pd.concat(results)

print(time.time()-t)

它只是运行而且没有做任何事情。在我的cmd窗口中，我在错误消息的末尾看到以下内容：

AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)>

在Anaconda上运行python版本3.6.1：

import sys
print(sys.version)
3.6.1 |Anaconda 4.4.0 (64-bit)| (default, May 11 2017, 13:25:24) [MSC v.1900 64 bit (AMD64)]

更新：仍然无法让它工作。下面是我运行提供的代码时的整个cmd日志。我很感激你的任何想法！

C:\Users\ywu\Google Drive>jupyter notebook
[I 10:13:43.954 NotebookApp] Serving notebooks from local directory: C:\Users\ywu\Google Drive
[I 10:13:43.954 NotebookApp] 0 active kernels
[I 10:13:43.955 NotebookApp] The Jupyter Notebook is running at: http://localhost:8888/?token=255a5c0c9af337a1c2187feb63f1c426fb903e5929a0b2f0
[I 10:13:43.956 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 10:13:43.959 NotebookApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://localhost:8888/?token=255a5c0c9af337a1c2187feb63f1c426fb903e5929a0b2f0
[I 10:13:44.264 NotebookApp] Accepting one-time-token-authenticated connection from ::1
[W 10:13:44.319 NotebookApp] 404 GET /api/kernels/aceb78ee-73e4-4481-9993-63e5ee8f72cb/channels?session_id=AEA3C6B2B0A440FC84FF3BAF5F5CB615 (127.0.0.1): Kernel does not exist: aceb78ee-73e4-4481-9993-63e5ee8f72cb
[W 10:13:44.328 NotebookApp] 404 GET /api/kernels/aceb78ee-73e4-4481-9993-63e5ee8f72cb/channels?session_id=AEA3C6B2B0A440FC84FF3BAF5F5CB615 (127.0.0.1) 20.07ms referer=None
[I 10:13:54.740 NotebookApp] Creating new notebook in /code/python
[I 10:13:55.241 NotebookApp] Kernel started: 45ab2da6-7466-408c-aa5a-98f7db54e711
[W 10:14:00.341 NotebookApp] Replacing stale connection: aceb78ee-73e4-4481-9993-63e5ee8f72cb:AEA3C6B2B0A440FC84FF3BAF5F5CB615
Process SpawnPoolWorker-2:
Traceback (most recent call last):
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 249, in _bootstrap
    self.run()
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
Process SpawnPoolWorker-1:
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\pool.py", line 108, in worker
    task = get()
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\queues.py", line 345, in get
    return _ForkingPickler.loads(res)
Process SpawnPoolWorker-4:
AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)>
Traceback (most recent call last):
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 249, in _bootstrap
    self.run()
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\pool.py", line 108, in worker
    task = get()
Traceback (most recent call last):
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\queues.py", line 345, in get
    return _ForkingPickler.loads(res)
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 249, in _bootstrap
    self.run()
Process SpawnPoolWorker-3:
AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)>
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
Process SpawnPoolWorker-5:
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\pool.py", line 108, in worker
    task = get()
Process SpawnPoolWorker-6:
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\queues.py", line 345, in get
    return _ForkingPickler.loads(res)
Traceback (most recent call last):
AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)>
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 249, in _bootstrap
    self.run()
Traceback (most recent call last):
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
Traceback (most recent call last):
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 249, in _bootstrap
    self.run()
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\pool.py", line 108, in worker
    task = get()
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 249, in _bootstrap
    self.run()
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\queues.py", line 345, in get
    return _ForkingPickler.loads(res)
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\pool.py", line 108, in worker
    task = get()
AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)>
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\pool.py", line 108, in worker
    task = get()
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\queues.py", line 345, in get
    return _ForkingPickler.loads(res)
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\queues.py", line 345, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)>
Process SpawnPoolWorker-7:
AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)>
Traceback (most recent call last):
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 249, in _bootstrap
    self.run()
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\pool.py", line 108, in worker
    task = get()
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\queues.py", line 345, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)>
Process SpawnPoolWorker-8:
Process SpawnPoolWorker-9:
Process SpawnPoolWorker-10:
Traceback (most recent call last):
Traceback (most recent call last):
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 249, in _bootstrap
    self.run()
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 249, in _bootstrap
    self.run()
Traceback (most recent call last):
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\pool.py", line 108, in worker
    task = get()
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 249, in _bootstrap
    self.run()
Process SpawnPoolWorker-11:
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\queues.py", line 345, in get
    return _ForkingPickler.loads(res)
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\pool.py", line 108, in worker
    task = get()
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)>
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\queues.py", line 345, in get
    return _ForkingPickler.loads(res)
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\pool.py", line 108, in worker
    task = get()
AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)>
Process SpawnPoolWorker-14:
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\queues.py", line 345, in get
    return _ForkingPickler.loads(res)
Traceback (most recent call last):
AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)>
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 249, in _bootstrap
    self.run()
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\pool.py", line 108, in worker
    task = get()
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\queues.py", line 345, in get
    return _ForkingPickler.loads(res)
Traceback (most recent call last):
AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)>
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 249, in _bootstrap
    self.run()
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\pool.py", line 108, in worker
    task = get()
  File "C:\Users\ywu\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\queues.py", line 345, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)>
[W 10:14:15.043 NotebookApp] 404 GET /api/kernels/c1224db6-69c6-470e-b74c-4c7b94fb48fe/channels?session_id=D8DC8A440B044EED8EBCA374EBEAF7C6 (127.0.0.1): Kernel does not exist: c1224db6-69c6-470e-b74c-4c7b94fb48fe
[W 10:14:15.046 NotebookApp] 404 GET /api/kernels/c1224db6-69c6-470e-b74c-4c7b94fb48fe/channels?session_id=D8DC8A440B044EED8EBCA374EBEAF7C6 (127.0.0.1) 7.48ms referer=None

Answer 1

经过大量研究后，似乎多处理不是在Windows上的笔记本中使用的选项。我正在关闭，但如果你有解决方案，请打开。我会切换到悲伤。

Answer 2

我在Windows上的Jupyter笔记本中进行了多处理，将我的功能保存在一个单独的.py文件中并将该文件包含在我的笔记本中。

示例：

<强> f.py：

def f(name, output):
  output.put('hello {0}'.format(name))
  return

Jupyter笔记本中的代码：

from multiprocessing import Process, Queue

#Having the function definition here results in
#AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>

#The solution seems to be importing the function from a separate file.

import f

#Also, the original version of f only had a print statement in it.  
#That doesn't work with Process - in the sense that it prints to the console 
#instead of the notebook.
#The trick is to let f write the string to print into an output-queue.
#When Process is done, the result is retrieved from the queue and printed.

if __name__ == '__main__':    

   # Define an output queue
   output=Queue()

   # Setup a list of processes that we want to run
   p = Process(target=f.f, args=('Bob',output))

   # Run process
   p.start()

   # Exit the completed process
   p.join()

   # Get process results from the output queue
   result = output.get(p)

   print(result)

我是一个Python新手，我可能错过了各种各样的细节，但这对我有用。

Answer 3

这个怎么样：

<强>代码：

#!/usr/bin/env python3

import sys
import time
import gender_guesser.detector as gender
import pandas as pd
import multiprocessing as mp

d = gender.Detector()

def guess_gender(name):
    n = name.title()
    g = d.get_gender(n)
    return g

def run():
    ls = ['john','joe','amamda','derick','peter','ashley','john',\
          'joe','amamda','derick','peter','ashley']

    num_cpus = mp.cpu_count() - 1
    pool = mp.Pool(processes=num_cpus)
    result = pool.map(guess_gender, ls)

    df = pd.DataFrame(result, columns=["gender"])

    print("\ntook {} secs to classify\n".format(str(time.time() - st)))
    print(df)  # or you could save the dataframe using .to_csv()

st = time.time()

if __name__ == "__main__":
    run()

<强>输出：

took 0.0150408744812 secs to classify

           gender
0            male
1            male
2         unknown
3            male
4            male
5   mostly_female
6            male
7            male
8         unknown
9            male
10           male
11  mostly_female

Answer 4

对于在Windows上使用Jupyter的人们来说，这个问题会让他们头疼。该代码可以在Linux系统上正常运行。

为了在Windows上运行代码，

1. 将功能定义放在单独的ipynb文件中。
2. 使用ipynb.fs.full.functions import func导入文件（确保先点子安装ipynb）
3. 这肯定可以解决这个问题。