我有以下代码:
val rawWeatherStream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topics)
val parsedWeatherStream: DStream[RawWeatherData] = ingestStream(rawWeatherStream)
我接受来自卡夫卡制作人的任何东西。然后将其发送到ingestStream
函数,该函数向我返回所有解析为RawWeatherData()
对象的对象。它的代码是:
def ingestStream(rawWeatherStream: InputDStream[(String, String)]): DStream[RawWeatherData] = {
val parsedWeatherStream = rawWeatherStream.map(_._2.split(","))
.map(RawWeatherData(_))
parsedWeatherStream
}
现在,我想使用平滑的方法将parsedWeatherStream
中的所有内容保存到DB2数据库中。我的架构是:
case class RawWeatherData(tag: Tag) extends Table[(String, Int, Int, Int, Int, Double, Double, Double, Int, Double, Int, String, Double, Double)](tag , "RawWeatherData") with WeatherModel{
var wsid = column[String]("wsid")
var year = column[Int]("year")
var month = column[Int]("month")
var day = column[Int]("day")
var hour = column[Int]("hour")
var temperature = column[Double]("temperature")
var dewpoint = column[Double]("depoint")
var pressure = column[Double]("pressure")
var windDirection = column[Int]("windDirection")
var windSpeed = column[Double]("windSpeed")
var skyCondition = column[Int]("skyCondition")
var skyConditionText = column[String]("skyConditionText")
var oneHourPrecip = column[Double]("oneHourPrecip")
var sixHourPrecip = column[Double]("sixHourPrecip")
def * = (wsid , year , month , day , hour , temperature , dewpoint , pressure , windDirection , windSpeed , skyCondition , skyConditionText , oneHourPrecip , sixHourPrecip)
}
要将数据保存到数据库中,我编写了以下代码:
val rawWeatherData = TableQuery[com.killrweather.data.RWD]
val db2 = Database.forConfig("db2")
val setup = DBIO.seq(
// Create the tables, including primary and foreign keys
rawWeatherData.schema.create,
// Insert some suppliers
parsedWeatherStream.foreachRDD { rawWeatherData += _._}
)
//parsedWeatherStream.foreachRDD { rawWeatherData += _._}
val setupFuture = db2.run(setup)
parsedWeatherStream.foreachRDD { rawWeatherData += _._}
部分代码是我尝试通过解析所有对象添加到数据库的地方,但是不幸的是,在该行代码中,出现以下错误Type mismatch, found: Unit, required: DBIOAction[_, NoStream, NotInferredE]
那么我现在如何使用Slick将来自Spark Streaming的数据插入数据库?