Question

我正在尝试实现python aiokafka异步库，由于某种原因，我无法异步处理消息

创建了异步消费者，生产者并使用了异步python库

环境：

 python 3.7.2
aiokafka==0.5.1
kafka-python==1.4.3

消费者：

from aiokafka import AIOKafkaConsumer
import asyncio
import json
import ast

loop = asyncio.get_event_loop()

async def consume():
    consumer = AIOKafkaConsumer(
        "test_topic", loop=loop, bootstrap_servers='localhost:9092')
    # Get cluster layout and topic/partition allocation
    await consumer.start()
    try:
        async for msg in consumer:
            sleep_time = ast.literal_eval(json.loads(msg.value))
            print('before sleep %s' % sleep_time)
            await asyncio.sleep(sleep_time)
            print('after sleep %s' % sleep_time)
    finally:
        await consumer.stop()

loop.run_until_complete(consume())

制作人：

import json
import uuid
from kafka import KafkaProducer, KafkaConsumer


class KafkaClient(object):
    def __init__(self, topic_name=None, consume=True):
        """
        Initial consumer and producer for Kafka
        :param topic_name: consumer topic name
        """
        self.topic_name = topic_name
        if topic_name is not None:
            self.kafka_connect(topic_name, source='SOURCE')

        self.producer = KafkaProducer(bootstrap_servers='localhost:9092',
                                      key_serializer=str.encode,
                                      value_serializer=lambda m: json.dumps(m).encode('utf-8'))


    def publish_message(self, topic_name, message, extra_data=None):
        try:
            msg_uid = str(uuid.uuid1())
            self.producer.send(topic_name, value=json.dumps(message))
            self.producer.flush()
            print('Message published [msg_uid]: %s' % msg_uid)
            return True

        except Exception as err:
            print(err)
            return False

k = KafkaClient()
for i in range(0, 1):
    k.publish_message('test_topic', 5)
    k.publish_message('test_topic', 3)
    k.publish_message('test_topic', 1)

希望通过此库对我的怀念：/

非常感谢您的帮助！：）

预期结果：

该过程将打印：

before sleep 5
before sleep 3
before sleep 1 
after sleep 1
after sleep 3
after sleep 5

实际结果：

过程打印

before sleep 5
after sleep 5
before sleep 3
after sleep 3
before sleep 1 
after sleep 1

Answer 1

async for本身不会并行处理序列-它仅允许协程暂停，同时等待异步可迭代对象产生下一个项目。您可以将其视为await特殊方法上的一系列__anext__，类似于普通的__next__调用。

但是很容易产生在消息到达时处理消息的任务。例如：

async def process(msg):
    sleep_time = ast.literal_eval(json.loads(msg.value))
    print('before sleep %s' % sleep_time)
    await asyncio.sleep(sleep_time)
    print('after sleep %s' % sleep_time)

async def consume():
    consumer = AIOKafkaConsumer(
        "test_topic", loop=loop, bootstrap_servers='localhost:9092')
    await consumer.start()
    tasks = []
    try:
        async for msg in consumer:
            tasks.append(asyncio.create_task(process(msg))
    finally:
        await consumer.stop()
    await asyncio.gather(*tasks)

Aiokafka库不是异步使用消息

环境：

消费者：

制作人：

预期结果：

实际结果：

1 个答案: