Question

KafkaStream如何使用主题消息。

以下是我的代码：

Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "my-app");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, KAFKA_SERVER_URL + ":" + KAFKA_SERVER_PORT);
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());

StreamsBuilder builder = new StreamsBuilder();
KafkaStreams streams = new KafkaStreams(builder.build(), props);
builder.stream(topic_name).print(null);

streams.start();

Answer 1

Kafka Streams DSL（特定域语言）建立在Streams Processor API的基础上。

它使用底层实现的底层处理器API来读取来自kafka主题的消息。这是详细的架构：

https://kafka.apache.org/20/documentation/streams/architecture

Streams DSL构建在Processor API之上。如果深入研究Processor API，您将看到如何实现功能以及如何使用一行代码轻松调用这些功能：

https://kafka.apache.org/20/documentation/streams/developer-guide/processor-api.html

这就是Stream DSL操作的工作方式。在使用Streams DSL编写KStream应用程序时，大多数操作可以在几行代码中调用，但是在其下面具有完整的实现

这是字数示例： https://github.com/confluentinc/kafka-streams-examples/blob/5.0.0-post/src/main/java/io/confluent/examples/streams/WordCountLambdaExample.java

最初，每个操作都将转换为 ProcessorNode 。因此，从主题读取将转换为 SourceNode ，而写入主题将是 SinkNode 。并将所有节点依次添加到拓扑中。
您可以在StreamsBuilder和StreamTask的源代码中看到更多详细信息。它将告诉您拓扑的构建和运行方式：

https://github.com/apache/kafka/blob/2.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java

以下是Wordcount的KStream应用程序示例。假设“ wordcount-input”是输入主题，而“ wordcount-output”是输出主题：

final String bootstrapServers = args.length > 0 ? args[0] : "localhost:9092";
final Properties streamsConfiguration = new Properties();
streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, "wordcount-lambda-example"); 
streamsConfiguration.put(StreamsConfig.CLIENT_ID_CONFIG, "wordcount-lambda-example-client");
streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); // add if you want to reset the offset to earliest for each run
final Serde<String> stringSerde = Serdes.String();
final Serde<Long> longSerde = Serdes.Long();
final StreamsBuilder builder = new StreamsBuilder();
final KStream<String, String> textLines = builder.stream("wordcount-input");
final Pattern pattern = Pattern.compile("\\W+", Pattern.UNICODE_CHARACTER_CLASS);
final KTable<String, Long> wordCounts = textLines
  .flatMapValues(value -> Arrays.asList(pattern.split(value.toLowerCase())))
  .groupBy((key, word) -> word)
  .count();
// Write the `KTable<String, Long>` to the output topic.
wordCounts.toStream().to("wordcount-output", Produced.with(stringSerde, longSerde));
final KafkaStreams streams = new KafkaStreams(builder.build(), streamsConfiguration);

streams.cleanUp();
streams.start();

1 个答案: