Question

我创建了一个包含4个分区的简单生产者，现在想要在一个使用者组中创建4个使用者来使用每个分区的数据。我怎么能这样做？

消费者代码

    public class KafkaConsumer {
    static List<String> list = new ArrayList<String>();
    public  static DataFrame reqFieldBOIDDf;
    public static DataFrame df ;
    static SparkConf conf = new SparkConf()

                     .setAppName("kafka-sandbox")

                     .setMaster("local[*]")

                     .set("spark.cassandra.connection.host","localhost"); //for cassandra

             static JavaSparkContext sc = new JavaSparkContext(conf);

    private static long lastOffset;
            public static void main(String[] str) throws InterruptedException {
               execute();
            }
            private static void execute() throws InterruptedException {

                    KafkaConsumer<String, String> consumer = createConsumer();       consumer.subscribe(Arrays.asList("KafkaConsumerTopic6"));
                    processRecords(consumer);
                    System.out.println("Inside execute");
            }
            private static KafkaConsumer<String, String> createConsumer() {
                    Properties props = new Properties();
                    props.put("bootstrap.servers", "localhost:9092");
                    String consumeGroup = "cg1";
                    props.put("group.id", consumeGroup);
                    props.put("enable.auto.commit", "true");
                    props.put("auto.commit.interval.ms", "101");
                    props.put("max.partition.fetch.bytes", "1035");
                    props.put("heartbeat.interval.ms", "3000");
                    props.put("session.timeout.ms", "6001");props.put("max.poll.records","500");
props.put("key.deserializer","org.apache.kafka.common.serialization.StringDeserializer");
                   props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
                    return new KafkaConsumer<String, String>(props);
            }
            private static void processRecords(KafkaConsumer<String, String> consumer) throws InterruptedException  {
                  while (true){
                           ConsumerRecords<String, String> records = consumer.poll(1000);
                            lastOffset = 0;
                            for (ConsumerRecord<String, String> record : records) {
                            lastOffset = record.offset();
                            list.add(record.value());
                             }
                   }
            }

一小段代码会非常有用。

提前致谢。

Answer 1

以与启动4位制作人相同的方式启动4位消费者。

确保它们都具有相同的group.id设置，并使它们全部订阅主题（如果它是4个主题，包含1个分区，则为主题）。由于他们将在同一组中，Kafka将自动为每个分区分配一个消费者。

具有4个分区的单生产者的多个消费者设置Kafka Java

1 个答案: