Question

我在尝试使用AVRO格式向KafkaIO apache Beam写消息时遇到以下错误。

Caused by: org.apache.kafka.common.errors.SerializationException: Error registering Avro schema: {"type":"record","name":"MyClass","namespace":"my.namespace","fields":[{"name":"value","type":"double"}]}
Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Schema being registered is incompatible with an earlier schema; error code: 409

管道只是从Kafka主题中读取相同的Avro消息，然后写入另一个主题。

要使用此消息，请执行以下操作：

Pipeline pipeline = Pipeline.create(options);

PTransform<PBegin, PCollection<KafkaRecord<String, MyClass>>> kafka_read = KafkaIO.<String, MyClass>read()
  .withBootstrapServers("localhost:9092")
  .withTopic("topic-in")
  .withKeyDeserializer(StringDeserializer.class)
  .withValueDeserializerAndCoder((Class) KafkaAvroDeserializer.class, AvroCoder.of(MyClass.class))
  .updateConsumerProperties(ImmutableMap.of("schema.registry.url", "http://localhost:8081"));

尝试写入Kafka时出现错误

pipeline.apply(kafka_read)
  .apply("Forward", ParDo.of(new TransformMyClass()))
  .apply(KafkaIO.<String, MyClass>write()
    .withBootstrapServers("localhost:9092")
    .withTopic("topic-out")
    .withKeySerializer(StringSerializer.class)
    .withValueSerializer((Class) KafkaAvroSerializer.class)
    .updateProducerProperties(ImmutableMap.of("schema.registry.url", "http://localhost:8081")));

MyClass是使用生产者和该使用者/生产者中的mvn generate-sources从架构创建的。

转换函数如下：

public class TransformMyClass extends DoFn<KafkaRecord<String, MyClass>, KV<String, MyClass>> {
@ProcessElement
public void transformMyClass(ProcessContext ctx) {
ctx.output(KV.of("key", ctx.element().getKV().getValue()));
}}

这是用KafkaIO编写Avro格式的正确方法吗？

谢谢。

Apache Beam Java KafkaIO编写Avro-正在注册的架构与早期的架构不兼容

0 个答案: