Question

请求来自龙卷风的Web应用程序的GET处理程序。从GET函数调用blocking_task函数。此blocking_task函数具有@run_on_executor装饰器。

但是这次执行失败了。你能帮忙吗？似乎motor db无法执行该线程。

import time
from concurrent.futures import ThreadPoolExecutor
from tornado import gen, web
from tornado.concurrent import run_on_executor
from tornado.ioloop import IOLoop
import argparse
from common.config import APIConfig
import sys
import os
import motor

parser = argparse.ArgumentParser()
parser.add_argument("-c", "--config-file", dest='config_file',
                    help="Config file location")
args = parser.parse_args()
CONF = APIConfig().parse(args.config_file)

client = motor.MotorClient(CONF.mongo_url)
db = client[CONF.mongo_dbname]
class Handler(web.RequestHandler):

    executor = ThreadPoolExecutor(10)

    def initialize(self):
        """ Prepares the database for the entire class """
        self.db = self.settings["db"]

    @gen.coroutine
    def get(self):
        self.blocking_task()

    @run_on_executor
    def blocking_task(self):
        mongo_dict = self.db.test_cases.find_one({"name": "Ping"})


if __name__ == "__main__":
    app = web.Application([
        (r"/", Handler),
    ],
        db=db,
        debug=CONF.api_debug_on,
    )
    app.listen(8888)
    IOLoop.current().start()




> ERROR:tornado.application:Exception in callback <functools.partial
> object at 0x7f72dfbe48e8> Traceback (most recent call last):   File
> "/usr/local/lib/python2.7/dist-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/ioloop.py",
> line 600, in _run_callback
>     ret = callback()   File "/usr/local/lib/python2.7/dist-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/stack_context.py",
> line 275, in null_wrapper
>     return fn(*args, **kwargs)   File "/usr/local/lib/python2.7/dist-packages/motor-0.5-py2.7.egg/motor/frameworks/tornado.py",
> line 231, in callback
>     child_gr.switch(future.result()) error: cannot switch to a different thread

你能帮忙吗？

Answer 1

来自docs

要使用的IOLoop和执行程序由io_loop和执行者自我属性。要使用不同的属性，请传递关键字装饰者的参数

您必须提供init threadpoolexecutor：

import time
from concurrent.futures import ThreadPoolExecutor
from tornado import gen, web
from tornado.concurrent import run_on_executor
from tornado.ioloop import IOLoop


class Handler(web.RequestHandler):

    executor = ThreadPoolExecutor(10)

    @gen.coroutine
    def get(self):
        self.blocking_task()

    @run_on_executor
    def blocking_task(self):
        time.sleep(10)


if __name__ == "__main__":
    app = web.Application([
        (r"/", Handler),
    ])
    app.listen(8888)
    IOLoop.current().start()

默认run_on_executor在executor属性中搜索threadpool，除非您明确传递其他内容，例如

_thread_pool = ThreadPoolExecutor(10)

@run_on_executor(executor='_thread_pool')
def blocking_task(self):
    pass

修改

基本上IOLoop应该用在单线程env中（你可以在每个线程上运行单独的IOLoop，但不是你的情况）。要与IOLoop通信，您应该使用add_callback，这是唯一的线程安全功能。

您可以使用：

@run_on_executor def blocking_task(self): IOLoop.instance().add_callback(some_update) @gen.coroutine def some_update(): db.test_cases.update({ "name": "abc" }, { "$set": { "status" : "xyz" } } )

但是你真的需要线程吗？如果您在main - IOLoop的线程上安排更新，那么单独线程的目的是什么。

Answer 2

最后关注代码，谢谢@kwarunek 还为回调函数添加了参数。

import time
from concurrent.futures import ThreadPoolExecutor
from tornado import gen, web
from tornado.concurrent import run_on_executor
from tornado.ioloop import IOLoop
import argparse
from common.config import APIConfig
import sys
import os
import motor

parser = argparse.ArgumentParser()
parser.add_argument("-c", "--config-file", dest='config_file',
                    help="Config file location")
args = parser.parse_args()
CONF = APIConfig().parse(args.config_file)

client = motor.MotorClient(CONF.mongo_url)
db = client[CONF.mongo_dbname]

class Handler(web.RequestHandler):

    executor = ThreadPoolExecutor(10)

    def initialize(self):
        """ Prepares the database for the entire class """
        self.db = self.settings["db"]

    @gen.coroutine
    def get(self):
        self.blocking_task("Ping", "Void-R")


    @run_on_executor
    def blocking_task(self, name, status):
        IOLoop.instance().add_callback(callback=lambda: self.some_update(name, status))

    @gen.coroutine
    def some_update(self, name, status):
        mongo_dict = yield self.db.test_cases.find_one({"name": name})
        self.db.test_cases.update({ "name": name }, { "$set": { "status" : status } } )


if __name__ == "__main__":
    app = web.Application([
        (r"/", Handler),
    ],
        db=db,
        debug=CONF.api_debug_on,
    )
    app.listen(8888)
    IOLoop.current().start()

Answer 3

Motor是一个非阻塞库，旨在从单个IOLoop线程中使用。您可以将ThreadPoolExecutor与PyMongo等阻塞库一起使用，但不得将其他线程与Motor配合使用。

相反，您应该直接使用yield调用Motor方法：

@gen.coroutine
def get(self):
    yield self.non_blocking_task()

@gen.coroutine
def non_blocking_task(self):
    motor_dict = yield self.db.test_cases.find_one({"name": "Ping"})

另请注意，如果您将@run_on_executor与PyMongo等阻塞库一起使用，装饰器会使阻塞函数无阻塞，因此必须使用yield调用修饰函数。

龙卷风错误：无法切换到其他线程

3 个答案: