Question

我有一个SchemaRegistry和一个KafkaBroker，可以使用Avro v1.8.1从中提取数据。对于反序列化，我一直在使用Confluent的KafkaAvroDeserializer。现在，我打算重构代码以使用Alpakka提供的Elasticsearch API，但不幸的是，这会破坏反序列化，因为它会导致NullPointerExceptions：

线程“ main”中的异常org.apache.kafka.common.errors.SerializationException：在偏移量0处反序列化分区topic-0的键/值时出错。如果需要，请查找记录以继续使用。引起原因：org.apache.kafka.common.errors.SerializationException：反序列化ID为2的Avro消息时出错造成原因：java.lang.NullPointerException 在io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.deserialize（AbstractKafkaAvroDeserializer.java:116）在io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.deserialize（AbstractKafkaAvroDeserializer.java:88）在io.confluent.kafka.serializers.KafkaAvroDeserializer.deserialize（KafkaAvroDeserializer.java:55）在org.apache.kafka.common.serialization.Deserializer.deserialize（Deserializer.java:58）在org.apache.kafka.clients.consumer.internals.Fetcher.parseRecord（Fetcher.java:1030）在org.apache.kafka.clients.consumer.internals.Fetcher.access $ 3300（Fetcher.java:110）在org.apache.kafka.clients.consumer.internals.Fetcher $ PartitionRecords.fetchRecords（Fetcher.java:1250）在org.apache.kafka.clients.consumer.internals.Fetcher $ PartitionRecords.access $ 1400（Fetcher.java:1099）在org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords（Fetcher.java:545）在org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords（Fetcher.java:506）在org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches（KafkaConsumer.java:1269）在org.apache.kafka.clients.consumer.KafkaConsumer.poll（KafkaConsumer.java:1200）在org.apache.kafka.clients.consumer.KafkaConsumer.poll（KafkaConsumer.java:1176）在de.adesso.fds.connectors.dpa.news.NewsConsumer.main（MyConsumer.java:58）

我一直按照example中所述使用Alpakka的ConsumerSettings API：

val system = ActorSystem.create();

// necessary to convert timestamps correctly in Avro Version 1.8.1 to avoid ClassCastExceptions
SpecificData.get().addLogicalTypeConversion(new TimeConversions.TimestampConversion());

val consumerSettings = ConsumerSettings.create(system, new StringDeserializer(), new KafkaAvroDeserializer())
    .withBootstrapServers(kafkaBootstrapServerUrl)
    .withClientId(InetAddress.getLocalHost().getHostName())
    .withGroupId("" + new Random().nextInt())
    .withProperty(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl)
    .withProperty(KafkaAvroDeserializerConfig.SPECIFIC_AVRO_READER_CONFIG, "true")
    .withProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
    .withStopTimeout(Duration.ofSeconds(5));

这些设置会导致NullPointerExceptions，而此香草Kafka Consumer道具可以正常工作：

val props = new Properties();
props.put(ConsumerConfig.CLIENT_ID_CONFIG, InetAddress.getLocalHost().getHostName()); 
props.put(ConsumerConfig.GROUP_ID_CONFIG, "" + new Random().nextInt());
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServerUrl);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
// necessary to convert timestamps correctly in newer Avro Versions and to avoid ClassCastExceptions
SpecificData.get().addLogicalTypeConversion(new TimeConversions.TimestampConversion());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, KafkaAvroDeserializer.class);
props.put(KafkaAvroDeserializerConfig.SPECIFIC_AVRO_READER_CONFIG, true);
props.put(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
val consumer = new KafkaConsumer<String, MyClass>(props);

在工作示例中，ConsumerRecords的值已成功反序列化为由AvroMavenPlugin从架构生成的类。

任何提示都值得赞赏！

Answer 1

我认为您需要将new KafkaAvroDeserializer()拉到其自己的变量，然后在该实例上调用.configure()方法以传递非空的注册表URL。

然后将已配置的实例传递到ConsumerSettings.create

FWIW，根据您的需求，Kafka Connect可以很好地加载Elasticsearch

在Alpakka上使用KafkaAvroDeserializer

1 个答案: