将流数据保存到表cassandra中,并带有一些主题名称

时间:2017-07-22 23:04:33

标签: spark-cassandra-connector

我有几个主题kafka的流数据,我想将每行RDD保存到特定的表cassandra中,我的RDD是名为 Stock 的案例类的集合:

Stock(test1,2017/07/23 00:01:02,14,Status)
Stock(test1,2017/07/23 00:01:03,78,Status)
Stock(test2,2017/07/23 00:01:02,86,Status)
Stock(test2,2017/07/23 00:01:03,69,Status)
Stock(test3,2017/07/23 00:01:02,46,Status)
Stock(test3,2017/07/23 00:01:03,20,Status)

我想获得此RDD中每行的第一个元素,它代表我的主题的名称,并在 saveToCassandra

中将其作为表的名称

任何想法请这是我没有我想要的代码

val messages = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topicsSet)
    .map(_._2)
messages.foreachRDD(rdd => {

                    val stockParsed = rdd.map(line => line.split(','))
                    .map(s => new Stock(s(0).toString, s(1).toString, s(2).toString, s(3).toString))

                    //here i want to give the fisrt element as name of table 
                    // if i have Stock(test1,2017/07/23 00:01:02,14,Status)  table = test1
                    // if i have Stock(test2,2017/07/23 00:01:02,14,Status)  table = test2
                    .saveToCassandra("sparkcassandra", table, SomeColumns("topic_name", "date_time", "mesure",                      "status"))
        })       
         ssc.start()
         ssc.awaitTermination()

    }
}  

0 个答案:

没有答案