Question

我正在尝试根据父级中记录的内容从主题（父级）写入kafka中的另一个主题（子级）。如果我从父主题使用的示例记录是{"date":{"string":"2017-03-20"},"time":{"string":"20:04:13:563"},"event_nr":1572470,"interface":"Transaction Manager","event_id":5001,"date_time":1490040253563,"entity":"Transaction Manager","state":0,"msg_param_1":{"string":"ISWSnk"},"msg_param_2":{"string":"Application startup"},"msg_param_3":null,"msg_param_4":null,"msg_param_5":null,"msg_param_6":null,"msg_param_7":null,"msg_param_8":null,"msg_param_9":null,"long_msg_param_1":null,"long_msg_param_2":null,"long_msg_param_3":null,"long_msg_param_4":null,"long_msg_param_5":null,"long_msg_param_6":null,"long_msg_param_7":null,"long_msg_param_8":null,"long_msg_param_9":null,"last_sent":{"long":1490040253563},"transmit_count":{"int":1},"team_id":null,"app_id":{"int":4},"logged_by_app_id":{"int":4},"entity_type":{"int":3},"binary_data":null}。

我想使用 entity 的值来写一个与实体值同名的主题（有一个固定数量的实体值，所以我可以静态创建，如果以编程方式动态创建主题很困难。我正在尝试使用这个

import org.apache.kafka.common.serialization.Serde;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.KeyValue;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KStreamBuilder;
import java.util.Properties;

public class entityDataLoader {
    public static void main(final String[] args) throws Exception {
    final Properties streamsConfiguration = new Properties();
    streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, "map-function-lambda-example");
    streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    streamsConfiguration.put(StreamsConfig.KEY_SERDE_CLASS_CONFIG, Serdes.ByteArray().getClass().getName());
    streamsConfiguration.put(StreamsConfig.VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());

    // Set up serializers and deserializers, which we will use for overriding the default serdes
    // specified above.
    final Serde<String> stringSerde = Serdes.String();
    final Serde<byte[]> byteArraySerde = Serdes.ByteArray();

    // In the subsequent lines we define the processing topology of the Streams application.
    final KStreamBuilder builder = new KStreamBuilder();

    // Read the input Kafka topic into a KStream instance.
    final KStream<byte[], String> textLines = builder.stream(byteArraySerde, stringSerde, "postilion-events");

    String content = textLines.toString();
    String entity = JSONExtractor.returnJSONValue(content, "entity");
    System.out.println(entity);

    textLines.to(entity);

    final KafkaStreams streams = new KafkaStreams(builder, streamsConfiguration);
    streams.cleanUp();
    streams.start();

    // Add shutdown hook to respond to SIGTERM and gracefully close Kafka Streams
    Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
    }
}

内容的内容为org.apache.kafka.streams.kstream.internals.KStreamImpl@568db2f2，很明显 @ KStream.toString（）不是正确的方法，可用于尝试获取实体价值。

P.S。 JSONExtractor类定义为

import org.json.simple.JSONObject;
import org.json.simple.parser.ParseException;
import org.json.simple.parser.JSONParser;
class JSONExtractor {

public static String returnJSONValue(String args, String value){
    JSONParser parser = new JSONParser();
    String app= null;
    System.out.println(args);
    try{
        Object obj = parser.parse(args);
        JSONObject JObj = (JSONObject)obj;
        app= (String) JObj.get(value);
        return app;
    }
    catch(ParseException pe){
        System.out.println("No Object found");
        System.out.println(pe);
    }
    return app;
}
}

Answer 1

您可以使用branch()将您的父流拆分为＆＃34;子流＆＃34;并写下每个＆＃34;子流＆＃34;一个输出主题（参见http://docs.confluent.io/current/streams/developer-guide.html#stateless-transformations）

您的branch()必须创建一个＆＃34;子流＆＃34;对于你输出的所有主题，但因为你知道所有主题，这应该不是问题。

此外，对于Kafka Streams，建议您在开始申请之前创建所有输出主题（参见http://docs.confluent.io/current/streams/developer-guide.html#user-topics）

使用kafkastreams根据记录内容的内容写入kafka主题

1 个答案: