我正在两个简单的KTable之间进行连接,假设我有2个主题,包含以下数据,请参阅详细信息:
第一个主题
1 | Victor C.
1 |瓦迪姆C.
2 | Vasile P.
3 | Vitalie C.
4 | Oleg C。
第二个主题
1 |程序员
2 |管理员
3 |管理器
当我执行以下查询时,SQL行为显然很明显,并且输出对我来说是明白的:
SELECT * FROM firstTable
INNER JOIN secondTable
ON firstTable.ID = secondTable.ID
1 | Victor C. | 1 |程序员
1 |瓦迪姆C. | 1 |程序员
2 | Vasile P. | 2 |管理员
3 | Vitalie C. | 3 |管理器
所以,我正在玩Kafka,我尝试做同样的行为,但加入的流的结果完全混淆了我的想法
请参阅代码段详细信息:
@Test
public void joinKTableToKTableWhereKeyValueIsIntegerAndString() throws Exception {
InternalTestConfiguration itc = getInternalTestConfiguration();
List < String > topics = Arrays.asList(itc.getFirstTopic(), itc.getSecondTopic(), itc.getProjectionTopic(), itc.getFirstKeyedTopic(), itc.getSecondKeyedTopic());
KafkaStreams streams = null;
try {
Integer partitions = 1;
Integer replication = 1;
RestUtils.createTopics(topics, partitions, replication, new Properties());
List < KeyValue < Integer, String >> employees = Arrays.asList(
new KeyValue < > (1, "Victor C."),
new KeyValue < > (1, "Vadim C."),
new KeyValue < > (2, "Vasile P."),
new KeyValue < > (3, "Vitalie C."),
new KeyValue < > (4, "Oleg C.")
);
List < KeyValue < Integer, String >> specialities = Arrays.asList(
new KeyValue < > (1, "Programmer"),
new KeyValue < > (2, "Administrator"),
new KeyValue < > (3, "Manager")
);
List < KeyValue < Integer, String >> expectedResults = Arrays.asList(
new KeyValue < > (1, "Victor C./Programmer"),
new KeyValue < > (1, "Vadim C./Programmer"),
new KeyValue < > (2, "Vasile P./Administrator"),
new KeyValue < > (3, "Vitalie C../Manager")
);
final Serde < Integer > keySerde = Serdes.Integer();
final Serde < String > valueSerde = Serdes.String();
Properties streamsConfiguration = new Properties();
streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, itc.getAppIdConfig());
streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, BOOTSTRAP_SERVERS_CONFIG);
streamsConfiguration.put(StreamsConfig.ZOOKEEPER_CONNECT_CONFIG, ZOOKEEPER_CONNECT_CONFIG);
streamsConfiguration.put(StreamsConfig.KEY_SERDE_CLASS_CONFIG, Serdes.Integer().getClass().getName());
streamsConfiguration.put(StreamsConfig.VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 10 * 1000);
//streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0);
streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory().getAbsolutePath());
KStreamBuilder builder = new KStreamBuilder();
KTable < Integer, String > firstKTable = builder.table(keySerde, valueSerde, itc.getFirstTopic(), itc.getFirstStore());
KTable < Integer, String > secondKTable = builder.table(keySerde, valueSerde, itc.getSecondTopic(), itc.getSecondStore());
KTable < Integer, String > projectionKTable = firstKTable.join(secondKTable, (l, r) - > {
return l + "/" + r;
});
projectionKTable.to(keySerde, valueSerde, itc.getProjectionTopic());
streams = new KafkaStreams(builder, streamsConfiguration);
streams.start();
Properties cfg1 = new Properties();
cfg1.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, BOOTSTRAP_SERVERS_CONFIG);
cfg1.put(ProducerConfig.ACKS_CONFIG, "all");
cfg1.put(ProducerConfig.RETRIES_CONFIG, 0);
cfg1.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, IntegerSerializer.class);
cfg1.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
IntegrationTestUtils.produceKeyValuesSynchronously(itc.getFirstTopic(), employees, cfg1);
Properties cfg2 = new Properties();
cfg2.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, BOOTSTRAP_SERVERS_CONFIG);
cfg2.put(ProducerConfig.ACKS_CONFIG, "all");
cfg2.put(ProducerConfig.RETRIES_CONFIG, 0);
cfg2.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, IntegerSerializer.class);
cfg2.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
IntegrationTestUtils.produceKeyValuesSynchronously(itc.getSecondTopic(), specialities, cfg2);
Properties consumerConfig = new Properties();
consumerConfig.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, BOOTSTRAP_SERVERS_CONFIG);
consumerConfig.put(ConsumerConfig.GROUP_ID_CONFIG, itc.getGroupIdConfig());
consumerConfig.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
consumerConfig.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class);
consumerConfig.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
List < KeyValue < Integer, String >> actualResults = IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(consumerConfig, itc.getProjectionTopic(), expectedResults.size());
assertThat(actualResults).containsExactlyElementsOf(expectedResults);
} finally {
if (streams != null) {
streams.close();
}
RestUtils.deleteTopics(topics);
}
}
我希望得到与SQL相同的结果,但事实并非如此。
//streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG,0)&#34;的结果停用
1,Vadim C./Programmer
2,Vasile P./Administrator
3,Vitalie C./Manager
1,Vadim C./Programmer
2,Vasile P./Administrator
3,Vitalie C./Manager
使用streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG,0)的结果;启用
1,Vadim C./Programmer
2,Vasile P./Administrator
3,Vitalie C./Manager
无论如何这两个结果都与SQL不一样,请帮助我理解这一点,因为我已经杀了我自己:(
答案 0 :(得分:2)
在您的SQL比较中,该数字似乎不是主键,因为Victor C.
和Vadim C.
都与1
如果数字是邮件的关键字,那么这在KTable中不起作用 - Vadim C.
会覆盖Victor C.
。这就是为什么你在输出中只有三个不同的人。
关于KTables缓存行为的问题的第二部分。启用缓存(第一个示例)后,在刷新缓存时会触发连接(默认情况下为30秒)。启用缓存时,还有an issue重复项。当您禁用缓存时,这不会发生,因此正确的&#34;正确&#34;输出没有重复。
我最近blogged about join behaviour in Kafka 0.10.1(所以不是最新版本改变了一些语义)。也许这对你有帮助。