来自两个分区的Kafka消费者

Question

在kafka中，我需要使用Java使用来自两个使用者的两个分区（分区1到使用者1，分区2到使用者2）使用一个主题。

这是我的生产者代码

public class KafkaClientOperationProducer {

    KafkaClientOperationConsumer kac = new KafkaClientOperationConsumer();

    public void initiateProducer(ClientOperation clientOperation,
        ClientOperationManager activityManager,Logger logger) throws Exception {


    Properties props = new Properties();
    props.put("bootstrap.servers","localhost:9092,localhost:9093,localhost:9094");
    props.put("key.serializer","org.apache.kafka.common.serialization.StringSerializer");
    props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

    Producer<String, ClientOperation> producer = new KafkaProducer<>(props);

    try{
    ProducerRecord<String, ClientOperation> record = new ProducerRecord<String, ClientOperation>(
            topicName, key, clientOperation);

    producer.send(record);

    }
    finally{
    producer.flush();
    producer.close();
    kac.initiateConsumer(activityManager);//Calling Consumer
    }
}
}

这是我的消费者代码

    public class KafkaClientOperationConsumer{

    String topicName = "CA_Topic";
    String groupName = "CA_TopicGroup";
    public  void initiateConsumer(ClientOperationManager activityManager) throws Exception {



    Properties props = new Properties();
    props.put("bootstrap.servers", "localhost:9092,localhost:9093,localhost:9094");
    props.put("group.id", groupName);
    props.put("enable.auto.commit", "true");
    props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
    props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

    KafkaConsumer<String, ClientOperation> consumer = new KafkaConsumer<>(props);
    consumer.subscribe(Arrays.asList(topicName));               

        ConsumerRecords<String, ClientOperation> records = consumer.poll(100);
        try{
        for (ConsumerRecord<String, ClientOperation> record : records) {    
            activityManager.save(record.value());//saves data in database

        }}
    finally{
        consumer.close();}
    }
    }

上面的代码对单个消费者有效，而不对多个消费者有效
clientOperation是一个对象，用于保存有关客户端操作的数据。
    分区号是三个（您可以从代码中看到），
当我尝试使用线程即（ExecutorService executor）调用initialConsumer时，数据库中出现了重复值     请更改代码，以便我可以使用两个使用者使用CA_Topic，由于内存问题，我不能使用两个JVM。在此先感谢

Answer 1

我猜您必须使用KafkaConsumer.assign方法。这里有个例子：

Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "ip:port");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "group_id");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.ByteArrayDeserializer");
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.ByteArrayDeserializer");

final Consumer<byte[], byte[]> consumer = new KafkaConsumer<>(props);
TopicPartition topicPartition = new TopicPartition("topic", 0); // topic name and partition id to be assigned for this consumer. in other consumer configurations this value must be any value other than 0
List<TopicPartition> partitionList = new ArrayList<TopicPartition>();
partitionList.add(topicPartition);
consumer.assign(partitionList); // in this line, 0. partition assigning to this consumer

您可以在Kafka的文档中查看详细信息：https://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#assign(java.util.Collection)