使用异步服务器的长时间运行任务

时间:2016-02-12 06:01:54

标签: python websocket tornado python-asyncio aiohttp

我想每个人都知道如何处理django中长期运行的任务:使用芹菜和放松。但是如果我想通过aiohttp(或龙卷风)获得websockets的好处呢?

假设我有很多CPU绑定任务,可能需要几秒钟到多个(5-10)分钟。在websocket循环中处理此任务并通知用户进度似乎是个好主意。没有ajax请求,对短任务的响应非常快。

async def websocket_handler(request):
    ws = web.WebSocketResponse()
    await ws.prepare(request)

    async for msg in ws:
        if msg.tp == aiohttp.MsgType.text:     
            answer_to_the_ultimate_question_of_life_the_universe_and_everything =\
                long_running_task(msg.data, NotificationHelper(ws))
            ws.send_str(json.dumps({
                'action': 'got-answer',
                'data': answer_to_the_ultimate_question_of_life_the_universe_and_everything,
            }))
    return ws

但另一方面,按照我的理解,以这种方式提供服务的CPU绑定任务会阻塞整个线程。如果我有10名工作人员和11名想要使用应用程序的客户,则在完成第一个客户的任务之前,将不会提供第11个客户端。

也许,我应该在celery中运行看起来很大的任务,在主循环中运行看起来很小的任务

所以,我的问题是:是否有任何良好的设计模式用于为异步服务器提供长期运行的任务?

谢谢!

1 个答案:

答案 0 :(得分:8)

Just run your long-running CPU-bound task by <iframe height="400" width="600" id="multGame" src="http://www.learninggamesforkids.com/_games/Main.swf?xml=assets/fill_in_blanks/fillitin_mult_six.xml"></iframe> <select id="timestableSlt" onchange="changeGame()"> <option value="1">1</option> <option value="2">2</option> <option value="3">3</option> <option value="4">4</option> <option value="5">5</option> <option value="6" selected="selected">6</option> <option value="7">7</option> <option value="8">8</option> <option value="9">9</option> <option value="10">10</option> <option value="11">11</option> <option value="12">12</option> </select> <script> function changeGame() { var e = document.getElementById("timestableSlt"); var strUser = e.options[e.selectedIndex].text; if (strUser = "1") { document.getElementById('multGame').src = "http://www.learninggamesforkids.com/_games/Main.swf?xml=assets/fill_in_blanks/fillitin_mult_one.xml"; } else if (strUser = "2") { document.getElementById('multGame').src = "http://www.learninggamesforkids.com/_games/Main.swf?xml=assets/fill_in_blanks/fillitin_mult_two.xml"; } else if (strUser = "3") { document.getElementById('multGame').src = "http://www.learninggamesforkids.com/_games/Main.swf?xml=assets/fill_in_blanks/fillitin_mult_three.xml"; } else if (strUser = "4") { document.getElementById('multGame').src = "http://www.learninggamesforkids.com/_games/Main.swf?xml=assets/fill_in_blanks/fillitin_mult_four.xml"; } else if (strUser = "5") { document.getElementById('multGame').src = "http://www.learninggamesforkids.com/_games/Main.swf?xml=assets/fill_in_blanks/fillitin_mult_five.xml"; } else if (strUser = "7") { document.getElementById('multGame').src = "http://www.learninggamesforkids.com/_games/Main.swf?xml=assets/fill_in_blanks/fillitin_mult_seven.xml"; } else if (strUser = "8") { document.getElementById('multGame').src = "http://www.learninggamesforkids.com/_games/Main.swf?xml=assets/fill_in_blanks/fillitin_mult_eight.xml"; } else if (strUser = "9") { document.getElementById('multGame').src = "http://www.learninggamesforkids.com/_games/Main.swf?xml=assets/fill_in_blanks/fillitin_mult_nine.xml"; } else if (strUser = "10") { document.getElementById('multGame').src = "http://www.learninggamesforkids.com/_games/Main.swf?xml=assets/fill_in_blanks/fillitin_mult_ten.xml"; } else if (strUser = "11") { document.getElementById('multGame').src = "http://www.learninggamesforkids.com/_games/Main.swf?xml=assets/fill_in_blanks/fillitin_mult_eleven.xml"; } else if (strUser = "12") { document.getElementById('multGame').src = "http://www.learninggamesforkids.com/_games/Main.swf?xml=assets/fill_in_blanks/fillitin_mult_twelve.xml"; } else { document.getElementById('multGame').src = "http://www.learninggamesforkids.com/_games/Main.swf?xml=assets/fill_in_blanks/fillitin_mult_six.xml"; } } </script> and send progress notifications by loop.run_in_executor().

If your job is not CPU but IO bound (sending emails for example) you may create a new task by loop.call_soon_threadsafe() call. It looks like spawning new thread.

If you cannot use fire-and-forget approach you need to use persistent message broker like RabbitMQ (there is https://github.com/benjamin-hodgson/asynqp library for communicating with Rabbit in asyncio way).