如何从spark消费者处理java中的传入流json数据

时间:2017-07-12 10:43:33

标签: postgresql apache-spark apache-kafka spark-streaming hadoop-streaming

下面是我的代码片段和流入消费者的JSON数据。但是我无法理解如何使用数据进行计算。

public void sparkKafkaConsumer(JavaStreamingContext jssc,String zkBroker,Map<String, Integer> topicmap, SQLContext sqlContext) throws InterruptedException  {
    System.out.println("INSIDE SPARK KAFKACONSUMER METHOD..........");


    System.out.println("Creating direct kafka stream with brokers and topics..........");
    // Create direct kafka stream with brokers and topics
    JavaPairReceiverInputDStream<String, String> messages = KafkaUtils.createStream(jssc, zkBroker,"CustomerKafkaConsumerThread", topicmap,StorageLevel.MEMORY_ONLY());

    System.out.println("Computation starts from here");
    messages.print();

以下是收到的JSON数据。

> (null,{"time":"2017/07/12
> 11:26:47","model":"20optky","speed":"20optky","cellId":"0605d822optky","course":"146.37optky","header":"ST600ALToptky","deviceId":"206675884optky","distance":"166optky","longitude":"-099.168493optky","latitude":"19.428616optky","payload":"ST600ALT+number+;206675884;20;376;20161005;16:26:59;0605d822;334;20;2ee5;63;+19.428616;-099.168493;000.213;146.37;6;1;166;12.21;000000;34;000887;4.4;1;0.00optky","date":"2017/07/12
> 11:26:47optky"})

请注意我使用的是Spark版本1.5.2,因此不支持DataFrame。

0 个答案:

没有答案