通过Apache Camel从Kafka消费Avro消息

时间:2015-04-20 12:51:31

标签: apache-camel unmarshalling apache-kafka

我有一个Apache Camel路由,将一条AVRO消息发布到Apache Kafka主题上。在设置producer属性' serializerClass = kafka.serializer.StringEncoder'时,我才能使用它。否则我得到

  

java.lang.ClassCastException:java.lang.String无法强制转换为[B     在kafka.serializer.DefaultEncoder.toBytes(Encoder.scala:34)at at   kafka.producer.async.DefaultEventHandler $$ anonfun $连载$ 1.适用(DefaultEventHandler.scala:130)     在   kafka.producer.async.DefaultEventHandler $$ anonfun $连载$ 1.适用(DefaultEventHandler.scala:125)     在   scala.collection.IndexedSeqOptimized $ class.foreach(IndexedSeqOptimized.scala:33)     在   scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)     在   kafka.producer.async.DefaultEventHandler.serialize(DefaultEventHandler.scala:125)     在   kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:52)     在kafka.producer.Producer.send(Producer.scala:77)at   kafka.javaapi.producer.Producer.send(Producer.scala:33)at   org.apache.camel.component.kafka.KafkaProducer.process(KafkaProducer.java:84)

另一方面,我有第二条Apache Camel路由应该从上面的主题消费

  

java.io.IOException:无效的长编码   org.apache.avro.io.BinaryDecoder.innerLongDecode(BinaryDecoder.java:217)     在org.apache.avro.io.BinaryDecoder.readLong(BinaryDecoder.java:176)     在   org.apache.avro.io.ResolvingDecoder.readLong(ResolvingDecoder.java:162)     在   org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)     在   org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)     在   org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)     在   org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)     在   org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)     在   org.apache.camel.dataformat.avro.AvroDataFormat.unmarshal(AvroDataFormat.java:133)     在   org.apache.camel.processor.UnmarshalProcessor.process(UnmarshalProcessor.java:67)

以下是我使用的Apache Camel消费者代码:

        <route id="cassandra.publisher">
            <from
                uri="{{kafka.base.uri}}&amp;topic=sensordata&amp;groupId=Cassandra_ConsumerGroup&amp;consumerId=CassandraConsumer_Instance_1&amp;clientId=adapter2" />      
            <unmarshal>
                <custom ref="avroSensorData" />
            </unmarshal>

2 个答案:

答案 0 :(得分:0)

为了解决这个问题,你必须为camel kafka使用者提供keyDeserializer和valueDeserializer,如下所示:

&amp; keyDeserializer = org.apache.kafka.common.serialization.StringDeserializer&amp; valueDeserializer = org.apache.kafka.common.serialization.ByteArrayDeserializer

答案 1 :(得分:-1)

http://camel.465427.n5.nabble.com/Camel-Kafka-Component-td5749525.html#a5769561

描述Apache Camel版本2.16.0 / 2.15.3将支持各种数据类型,而不仅仅是字符串消息。

正如所承诺的那样,已经使用Apache Camel 2.15.3进行了修复,并使用CAMEL-8790(https://issues.apache.org/jira/browse/CAMEL-8790)进行了修复。