如何在Spark Streaming中使用Kafka主题的自定义对象？

Question

我正在尝试使用Java API在Spark Streaming中使用来自Kafka集群的消息。我可以将消息用作String而不是自定义对象。

我的配置是：

  

producer- producerConfig.metadata.broker.list = 192.168.56.2：9092   producerConfig.zk.connect = 192.168.188.104：2181   producerConfig.serializer.class = kafka.serializer.StringEncoder   producerConfig.request.required.acks = 1

     

consumer- consumerConfig.master = local consumerConfig.cassandraHost =   192.168.188.104

public class SparkJavaDemo {
    public static void main(String arr[]) {
        String host = "10.206.132.27:2181";// zookeeper
        String group = "test-group";// consumer group name
        Map<String, Integer> consumeJourney = new HashMap<String, Integer>();
        consumeJourney.put("test", 5);// topic map --> topic-partition      
        SparkConf sparkConf = new SparkConf(true);
        sparkConf.setAppName("Sample");     
        sparkConf.setMaster("local");
        sparkConf.set("spark.cassandra.connection.host", "10.206.132.27");
        JavaSparkContext sc = new JavaSparkContext(sparkConf);
        JavaStreamingContext jssc = new JavaStreamingContext(sc, new Duration(
                3000));     
        JavaPairReceiverInputDStream<String, String> kafkaStream = KafkaUtils
                .createStream(jssc, host, group, consumeJourney);           
        JavaDStream<String> lines = kafkaStream.map(
                new Function<Tuple2<String, String>, String>() {
                    public String call(Tuple2<String, String> message) {
                        System.out.println("inside map"+message);
                        return message._2();
                    }
                }
            );  
        lines.print();  
        jssc.start();
        jssc.awaitTermination();
}

以上代码有效。 但是，我想将消息消费到Test类型的对象中。 所以，我尝试将javaDStream修改为：

JavaDStream<Test> lines = kafkaStream.map(new Function<Tuple2<String, String>,Test>() {             
            @Override
            public Test call(Tuple2<String, String> tuple2)
                    throws JSONException,Exception {                
                System.out.println("inside call()");    
                String jsonString = tuple2._2();
                System.out.println(jsonString);
                JSONObject record = (JSONObject) JSON.parse(jsonString);                
                Integer id = record.getInt("id");
                String name = record.getString("name");             
                return new Test(id,name);

            }
});

但它不起作用。 请帮忙。