Question

我正在使用Kafka v0.10.0.0并创建了Producer＆amp;消费者Java代码。但代码仍然停留在producer.send上，日志中没有任何异常。

任何人都可以帮忙。提前谢谢。

我正在使用/修改“mapr - kakfa示例程序”。你可以在这里查看完整的代码。 https://github.com/panwars87/kafka-sample-programs

**重要提示：我在maven依赖项中将kafka-client版本更改为0.10.0.0并在本地运行Kafka 0.10.0.0。

public class Producer {
public static void main(String[] args) throws IOException {
    // set up the producer
    KafkaProducer<String, String> producer;
    System.out.println("Starting Producers....");
    try (InputStream props = Resources.getResource("producer.props").openStream()) {
        Properties properties = new Properties();
        properties.load(props);
        producer = new KafkaProducer<>(properties);
        System.out.println("Property loaded successfully ....");
    }

    try {
        for (int i = 0; i < 20; i++) {
            // send lots of messages
            System.out.println("Sending record one by one....");
            producer.send(new ProducerRecord<String, String>("fast-messages","sending message - "+i+" to fast-message."));

            System.out.println(i+" message sent....");
            // every so often send to a different topic
            if (i % 2 == 0) {
                producer.send(new ProducerRecord<String, String>("fast-messages","sending message - "+i+" to fast-message."));
                producer.send(new ProducerRecord<String, String>("summary-markers","sending message - "+i+" to summary-markers."));
                producer.flush();
                System.out.println("Sent msg number " + i);
            }
        }
    } catch (Throwable throwable) {
        System.out.printf("%s", throwable.getStackTrace());
        throwable.printStackTrace();
    } finally {
        producer.close();
    }

  }
}

public class Consumer {
public static void main(String[] args) throws IOException {

    // and the consumer
    KafkaConsumer<String, String> consumer;
    try (InputStream props = Resources.getResource("consumer.props").openStream()) {
        Properties properties = new Properties();
        properties.load(props);
        if (properties.getProperty("group.id") == null) {
            properties.setProperty("group.id", "group-" + new Random().nextInt(100000));
        }
        consumer = new KafkaConsumer<>(properties);
    }
    consumer.subscribe(Arrays.asList("fast-messages", "summary-markers"));
    int timeouts = 0;
    //noinspection InfiniteLoopStatement
    while (true) {
        // read records with a short timeout. If we time out, we don't really care.
        ConsumerRecords<String, String> records = consumer.poll(200);
        if (records.count() == 0) {
            timeouts++;
        } else {
            System.out.printf("Got %d records after %d timeouts\n", records.count(), timeouts);
            timeouts = 0;
        }
        for (ConsumerRecord<String, String> record : records) {
            switch (record.topic()) {
                case "fast-messages":
                    System.out.println("Record value for fast-messages is :"+ record.value());            
                        break;
        case "summary-markers":
            System.out.println("Record value for summary-markers is :"+ record.value());
                        break;
                default:
                    throw new IllegalStateException("Shouldn't be possible to get message on topic ");
            }
        }
    }
   }
}

Answer 1

您正在运行的代码是用于mapR的演示，而不是Kafka。 MapR声称API与Kafka 0.9的兼容性，但即使这样，mapR也会以不同的方式处理消息偏移，而Kafka（偏移是消息的字节偏移而不是增量偏移）等等.mapR实现也至少可以说非常不同。这意味着如果你很幸运，Kafka 0.9应用程序可能恰好在mapR上运行，反之亦然。其他版本没有这样的保证。

Answer 2

感谢大家的所有投入。我通过调整Mapr代码并引用其他一些帖子来解决这个问题。解决方案api的链接：

https://github.com/panwars87/hadoopwork/tree/master/kafka/kafka-api

Kafka Producer Consumer API问题

2 个答案: