在Tornado中的线程中运行长阻塞函数时遇到问题

时间:2013-10-11 19:37:50

标签: python multithreading tornado

我对龙卷风很新。刚刚看到如何处理在Tornado阻止的请求。我在一个单独的线程中运行阻塞代码。但是主线程仍会阻塞,直到线程函数完成。我这里没有使用gen.coroutine,但是已经尝试了,结果是相同的

counter = 0

def run_async(func):
    @wraps(func)
    def function_in_a_thread(*args, **kwargs):
        func_t = Thread(target=func, args=args, kwargs=kwargs)
        func_t.start()
    return function_in_a_thread

def long_blocking_function(index, sleep_time, callback):
    print "Entering run counter:%s" % (index,)
    time.sleep(sleep_time)
    print "Exiting run counter:%s" % (index,)

    callback('keyy' + index)


class FooHandler(tornado.web.RequestHandler):

    @web.asynchronous
    def get(self):

        global counter
        counter += 1
        current_counter = str(counter)

        print "ABOUT to spawn thread for counter:%s" % (current_counter,)
        long_blocking_function(
            index=current_counter,
            sleep_time=5, callback=self.done_waiting)


        print "DONE with the long function"

    def done_waiting(self, response):
        self.write("Whatever %s " % (response,))
        self.finish()


class Application(tornado.web.Application):
    def __init__(self):
        handlers = [(r"/foo", FooHandler),
                    ]

        settings = dict(
            debug=True,
        )

        tornado.web.Application.__init__(self, handlers, **settings)


def main():
    application = Application()
    application.listen(8888)
    tornado.ioloop.IOLoop.instance().start()

if __name__ == "__main__":
    main()

当我发出背靠背请求时,FooHandler会阻塞并且在long_blocking_function完成之前不会收到任何请求。所以我最终看到像

这样的东西
ABOUT to spawn thread for counter:1
Entering run counter:1
Exiting run counter:1
DONE with the long function
ABOUT to spawn thread for counter:2
Entering run counter:2
Exiting run counter:2
DONE with the long function
ABOUT to spawn thread for counter:3
Entering run counter:3
Exiting run counter:3
DONE with the long function

我期待这些内容(因为我在第一次调用long_blocking_function之前发出了多个请求),但我只看到类似于上面的跟踪

ABOUT to spawn thread for counter:1
DONE with the long function
ABOUT to spawn thread for counter:2
DONE with the long function
ABOUT to spawn thread for counter:3
DONE with the long function
ABOUT to spawn thread for counter:4
DONE with the long function

我看了Tornado blocking asynchronous requests并尝试了两种解决方案。但是当我用同一个处理程序的背靠背请求运行它们时,它们都阻塞了。有人能弄清楚我做错了什么吗?我知道龙卷风在多线程方面表现不佳,但我应该能够以非阻塞方式从中运行一个新线程。

3 个答案:

答案 0 :(得分:5)

Tornado可以很好地使用concurrent.futures库(可用Python 2.x backport),因此您可以使用ThreadPoolExecutor将长时间运行的请求移交给线程池。

这种技术非常有效 - 我们使用它来处理长时间运行的数据库操作。当然,在现实世界中,你也希望以健壮和优雅的方式处理超时和其他异常,但我希望这个例子足以说明这个想法。

def long_blocking_function(index, sleep_time, callback):
    print ("Entering run counter:%s" % (index,))
    time.sleep(sleep_time)
    print ("Exiting run counter:%s" % (index,))
    return "Result from %d" % index


class FooHandler(tornado.web.RequestHandler):
    @tornado.gen.coroutine
    def get(self):
        global counter
        counter += 1
        current_counter = str(counter)

        print ("ABOUT to spawn thread for counter:%s" % (current_counter,))
        result = yield self.executor.submit(long_blocking_function,
                                            index=current_counter,
                                            sleep_time=5)
        self.write(result)
        print ("DONE with the long function")

答案 1 :(得分:1)

当您将任务卸载到另一个线程时,在任务完成后调用的事件循环中注册一个回调。答案在这个帖子中:How to make a library asynchronous in python

问候 米。

答案 2 :(得分:0)

您只是忘了实际使用已定义的run_async装饰器。

import tornado.web
import functools
import threading
import time

counter = 0
def run_async(func):
    @functools.wraps(func)
    def function_in_a_thread(*args, **kwargs):
        func_t = threading.Thread(target=func, args=args, kwargs=kwargs)
        func_t.start()
    return function_in_a_thread


@run_async
def long_blocking_function(index, sleep_time, callback):
    print ("Entering run counter:%s" % (index,))
    time.sleep(sleep_time)
    print ("Exiting run counter:%s" % (index,))
    callback('keyy' + index)

class FooHandler(tornado.web.RequestHandler):
    @tornado.web.asynchronous
    def get(self):
        global counter
        counter += 1
        current_counter = str(counter)

        print ("ABOUT to spawn thread for counter:%s" % (current_counter,))
        long_blocking_function(
            index=current_counter,
            sleep_time=5, callback=self.done_waiting)
        print ("DONE with the long function")

    def done_waiting(self, response):
        self.write("Whatever %s " % (response,))
        self.finish()


class Application(tornado.web.Application):
    def __init__(self):
        handlers = [(r"/foo", FooHandler),
                    ]

        settings = dict(
            debug=True,
        )

        tornado.web.Application.__init__(self, handlers, **settings)


def main():
    application = Application()
    application.listen(8888)
    tornado.ioloop.IOLoop.instance().start()

if __name__ == "__main__":
    main()