Question

我目前有一种情况，我需要执行和监控一些异步工作，这些工作可能是大约 99% 的网络 IO 阻塞任务，需要尽快执行。为了实现并行性，我使用了 Python Multiprocessing 库。我编写了以下示例：

import asyncio
import multiprocessing
import random
from multiprocessing import Process
from multiprocessing.queues import Queue
from queue import Empty


class DefaultQueue(Queue):
    def get_nowait(self, default=None):
        try:
            return self.get(block=False)
        except Empty:
            return default


async def do_work(event, worker_nr):
    await asyncio.sleep(random.random())  # simulate network IO blocking task
    print(f"Worker {worker_nr} received: {event}")


async def queue_worker(queue, worker_nr):
    loop = asyncio.get_event_loop()
    while True:
        event = queue.get_nowait()
        if event:
            loop.create_task(do_work(event, worker_nr))
        await asyncio.sleep(0)


def run_queue_worker(queue, worker_nr):
    asyncio.run(queue_worker(queue, worker_nr))


def queue_publisher(queue):
    for i in range(100):
        queue.put(f'event: {i}')


if __name__ == '__main__':

    work_queue = DefaultQueue(ctx=multiprocessing.get_context())
    workers = [Process(target=run_queue_worker, args=(work_queue, i)) for i in range(4)]
    publisher = Process(target=queue_publisher, args=(work_queue,))

    processes = workers

    for w in processes + [publisher]:
        w.start()

    try:
        while all([p.is_alive() for p in processes]):
            pass
    finally:
        print(f"Sending terminal signal processes")
        for p in processes:
            p.terminate()

        for p in processes:
            p.join()

        print("Main process shutdown!")

但我对这个补丁有两个主要问题：

（错误）使用 asyncio.sleep 进行显式上下文切换感觉就像代码异味。
loop.create_task 基本上是将工作块的执行推迟到事件循环的下一个滴答声，这会增加延迟。如果可能，我想立即评估任务，而不是等待下一个滴答声。

虽然我感觉我理解异步构造的基本概念，但我不知道如何将这两个问题合并到最终设计中。有什么想法吗？

Python 多处理队列异步工作者

0 个答案: