Question

I need to setup Kafka producer to send 500 msgs at one batch not msg by msg but bulk import. I checked https://github.com/dpkp/kafka-python/issues/479 and tried producer.send_messages(topic, *message) but it fails with error:

>       producer.send_messages('event_connector_mt', *load_entries)
E       AttributeError: 'cimpl.Producer' object has no attribute 'send_messages'

I also tried to pass it like producer.produce(topic, *message) fails with:

       producer.produce('event_connector_mt', *load_entries)
E       TypeError: function takes at most 8 arguments (501 given)

So i dig more and found out that i have to set in producer configuration the type to be async and batch.size to be bigger than default, but when i try config like:

from confluent_kafka import Consumer, Producer       

producer = Producer(**{'bootstrap.servers': KAFKA_BROKERS,
                       'queue.buffering.max.messages': 1000000,
                       'batch.num.messages': 500,
                       'batch.size': 19999,
                       'producer.type': 'async'
                       })

fails with:

E       KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="No such configuration property: "producer.type""}

same error for batch.size Can you pint me where and how i can setup async and batch size or any other way to pass bulk msgs to Kafka 0.9.3.1

Answer 1

默认情况下，所有生产者都是异步的。底层librdkafka库不支持Producer.type和batch.size配置。

因此，请使用可用配置batch.num.messages或message.max.bytes。

Answer 2

你正在混淆Confluent python客户端名为＆＃34; confluent-kafka-python＆＃34;

http://docs.confluent.io/3.2.1/clients/confluent-kafka-python/

使用＆＃34; kafka-python＆＃34;客户端

https://github.com/dpkp/kafka-python

这是两个具有不同API的不同客户端。

Confluent Kafka Python library configure producer for bulk msg

2 个答案: