我有几个主题kafka的流数据,我想将每行RDD保存到特定的表cassandra中,我的RDD是名为 Stock 的案例类的集合:
Stock(test1,2017/07/23 00:01:02,14,Status)
Stock(test1,2017/07/23 00:01:03,78,Status)
Stock(test2,2017/07/23 00:01:02,86,Status)
Stock(test2,2017/07/23 00:01:03,69,Status)
Stock(test3,2017/07/23 00:01:02,46,Status)
Stock(test3,2017/07/23 00:01:03,20,Status)
我想获得此RDD中每行的第一个元素,它代表我的主题的名称,并在 saveToCassandra
中将其作为表的名称任何想法请这是我没有我想要的代码
val messages = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topicsSet)
.map(_._2)
messages.foreachRDD(rdd => {
val stockParsed = rdd.map(line => line.split(','))
.map(s => new Stock(s(0).toString, s(1).toString, s(2).toString, s(3).toString))
//here i want to give the fisrt element as name of table
// if i have Stock(test1,2017/07/23 00:01:02,14,Status) table = test1
// if i have Stock(test2,2017/07/23 00:01:02,14,Status) table = test2
.saveToCassandra("sparkcassandra", table, SomeColumns("topic_name", "date_time", "mesure", "status"))
})
ssc.start()
ssc.awaitTermination()
}
}