Question

我的django应用程序遇到问题，该应用程序使用随机森林分类器（http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html）对项目进行分类。我收到的错误是：

AttributeError at /items/

'Thread' object has no attribute '_children'

Request Method:     POST
Request URL:    http://localhost:8000/items/
Django Version:     1.7.6
Exception Type:     AttributeError
Exception Value:    

'Thread' object has no attribute '_children'

Exception Location:     /usr/lib/python2.7/multiprocessing/dummy/__init__.py in start, line 73
Python Executable:  /home/cristian/env/bin/python
Python Version:     2.7.3
Python Path:    

['/home/cristian/filters',
 '/home/cristian/env/lib/python2.7',
 '/home/cristian/env/lib/python2.7/plat-linux2',
 '/home/cristian/env/lib/python2.7/lib-tk',
 '/home/cristian/env/lib/python2.7/lib-old',
 '/home/cristian/env/lib/python2.7/lib-dynload',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-linux2',
 '/usr/lib/python2.7/lib-tk',
 '/home/cristian/env/local/lib/python2.7/site-packages']

Server time:    Fri, 24 Apr 2015 16:08:20 +0000

问题在于我根本不使用线程。这是代码：

def item_to_dict(item):
    item_dict = {}
    for key in item:
            value = item[key]
            # fix encoding 
            if isinstance(value, unicode):
                    value = value.encode('utf-8')
            item_dict[key] = [value]
    return item_dict

def load_classifier(filter_name):
        clf = joblib.load(os.path.join(CLASSIFIERS_PATH, filter_name, 'random_forest.100k.' + filter_name.lower() +  '.pkl'))
        return clf

@api_view(['POST'])
 def classify_item(request):
    """
    Classify item
    """
    if request.method == 'POST':
            serializer = ItemSerializer(data=request.data['item'])
            if serializer.is_valid():
                    # get item and filter_name
                    item = serializer.data
                    filter_name = request.data['filter']

                    item_dict = item_to_dict(item)

                    clf = load_classifier(filter_name)

                    # score item
                    y_pred = clf.predict_proba(pd.DataFrame(item_dict))
                    item_score = y_pred[0][1]

                    # create and save classification
                    classification = Classification(classifier_name=filter_name,score=item_score,item_id=item['_id'])
                    classification_serializer = ClassificationSerializer(classification)
                    return Response(classification_serializer.data, status=status.HTTP_201_CREATED)
            else:
                    return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

我能打印出＆＃34; clf＆＃34;和＆＃34; item_dict＆＃34;变量和一切似乎都可以。当我调用方法＆＃34; predict_proba＆＃34;分类器。要添加的一件重要事情是，当我第一次运行服务器并发送post方法时，我没有收到错误。

这里是完整的追溯：

File "/home/cristian/env/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
line 111.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/cristian/env/local/lib/python2.7/site-packages/django/views/decorators/csrf.py" in wrapped_view
line 57.         return view_func(*args, **kwargs)
File "/home/cristian/env/local/lib/python2.7/site-packages/django/views/generic/base.py" in view
line 69.             return self.dispatch(request, *args, **kwargs)
File "/home/cristian/env/local/lib/python2.7/site-packages/rest_framework/views.py" in dispatch
line 452.             response = self.handle_exception(exc)
File "/home/cristian/env/local/lib/python2.7/site-packages/rest_framework/views.py" in dispatch
line 449.             response = handler(request, *args, **kwargs)
File "/home/cristian/env/local/lib/python2.7/site-packages/rest_framework/decorators.py" in handler
line 50.             return func(*args, **kwargs)
File "/home/cristian/filters/classifiers/views.py" in classify_item
line 70.             y_pred = clf.predict_proba(pd.DataFrame(item_dict))
File "/home/cristian/env/local/lib/python2.7/site-packages/sklearn/pipeline.py" in predict_proba
line 159.         return self.steps[-1][-1].predict_proba(Xt)
File "/home/cristian/env/local/lib/python2.7/site-packages/sklearn/ensemble/forest.py" in predict_proba
line 468.             for i in range(n_jobs))
File "/home/cristian/env/local/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py" in __call__
line 568.             self._pool = ThreadPool(n_jobs)
File "/usr/lib/python2.7/multiprocessing/pool.py" in __init__
line 685.         Pool.__init__(self, processes, initializer, initargs)
File "/usr/lib/python2.7/multiprocessing/pool.py" in __init__
line 136.         self._repopulate_pool()
File "/usr/lib/python2.7/multiprocessing/pool.py" in _repopulate_pool
line 199.             w.start()
File "/usr/lib/python2.7/multiprocessing/dummy/__init__.py" in start
line 73.         self._parent._children[self] = None

Exception Type: AttributeError at /items/
Exception Value: 'Thread' object has no attribute '_children'

Answer 1

作为解决方法，您可以使用以下命令在预测时禁用线程：

clf = load_classifier(filter_name)
clf.set_params(n_jobs=1)
y_pred = clf.predict_proba(pd.DataFrame(item_dict))

另请注意，在每个请求中调用load_classifier可能会很昂贵，实际上会从磁盘加载模型。

您可以将mmap_mode='r'传递给joblib.load到内存映射磁盘中的数据。即使您同时访问相同的模型参数的并发请求（如果使用像gunicorn这样的不同线程和不同的Python进程），它也可以只加载模型一次。

Answer 2

看起来像that problem was fixed from Python 2.7.5。这基本上是multiprocessing中的一个错误。

＆＃39;主题＆＃39;对象没有属性＆＃39; _children＆＃39; - django + scikit-learn

2 个答案: