Question

我正在尝试创建一个应用程序，该应用程序将接收传入事件，并将IMEI编号与资产ID信息（在外部数据库中填充）结合在一起。我正在使用Kafka Connect将这些数据来源转换为Kafka主题，然后创建两个流，但是它似乎没有用，并且没有错误。我正在使用Avro对象。

我已经为两个流重新设置了密钥，以便可以使用IMEI进行连接。联接似乎无法正常工作，我几乎可以确定它是联接元素，因为当流式数据通过时，它将被打印到控制台，而联接的数据则无效。

    final Serde<personalInformation> assetEntryDBSpecificAvroSerde = new SpecificAvroSerde<>();
    assetEntryDBSpecificAvroSerde.configure(serdeConfig, false);

    final Serde<bornRecordReport> eventEntrySpecificAvroSerde = new SpecificAvroSerde<>();
    eventEntrySpecificAvroSerde.configure(serdeConfig, false);


    StreamsBuilder builder = new StreamsBuilder();

    //Streaming events from Kafka topic
    KStream<Long, bornRecordReport> eventEntriesStream = builder.stream(eventEntryStream, Consumed.with(stringSerde, eventEntrySpecificAvroSerde))
            .selectKey((key, value) -> value.getIMEI());

    eventEntriesStream.foreach((key,value) -> System.out.println("Event Stream: Key = " + key + ", Value = " + value));

    //Streaming Asset ID updates from asset ID topic
    KStream<Long, personalInformation> sourceAssetEntryStream = builder.stream(assetEntryDBStream, Consumed.with(stringSerde, assetEntryDBSpecificAvroSerde))
            .selectKey((key, value) -> value.getIMEI());                                       

    sourceAssetEntryStream.foreach((key, value) -> System.out.println("Asset Stream: Key = " + key + ", Value = " + value));

    //Joining the two streams
    KStream<Long, bornRecordPersonalInformation> joinedStreams = eventEntriesStream.join(sourceAssetEntryStream, (eventVal, assetVal) -> bornRecordPersonalInformation.newBuilder()
                .setAssetID(assetVal.getAssetID())
                .setIMEI(eventVal.getIMEI())
                .setRemoteEventUID(eventVal.getRemoteEventUID())
                .setReceivedPacketUID(eventVal.getReceivedPacketUID())
                .build(),
             JoinWindows.of(TimeUnit.SECONDS.toSeconds(100)),
             Joined.with(longSerde, eventEntrySpecificAvroSerde, assetEntryDBSpecificAvroSerde));

    joinedStreams.peek((key, value) -> System.out.println("Asset Association Stream: Key = " + key + ", Value = " + value));
    joinedStreams.to(joinedStreamsStream);

    Topology topology = builder.build();
    KafkaStreams streams = new KafkaStreams(topology, createProperties());
    streams.start();

Avro对象：

{
"type":"record",
    "namespace":"com.-.kafkafrontend",
"name":"personalInformation",
"fields": [
    { "name":"IMEI", "type":"long", "doc":"Unique Event ID"},
    { "name":"assetID", "type":"long", "doc":"Unique Asset IDeve"}
    ]
}
{
"type":"record",
"namespace":"com.-.kafkafrontend",
"name":"bornRecordReport",
"fields": [
    { "name":"remoteEventUID", "type":"long", "doc":"Unique Event ID"},
    { "name":"receivedPacketUID", "type":"long", "doc":"Unique Packet ID"},
    { "name":"IMEI", "type":"long", "doc":"'0' - '9'"}
    ]
}
{
"type":"record",
"namespace":"com.-.kafkafrontend",
"name":"bornRecordPersonalInformation",
"fields": [
    { "name":"remoteEventUID", "type":"long", "doc":"Unique Event ID"},
    { "name":"receivedPacketUID", "type":"long", "doc":"Unique Packet ID"},
    { "name":"IMEI", "type":"long", "doc":"'0' - '9'"},
    { "name":"assetID", "type":"long", "doc":"Unique Asset ID"}
    ]
 }

如何使用Kafka加入两个流

0 个答案: