使用Slick将Spark Streaming中的批量数据插入数据库

时间:2019-04-28 06:35:40

标签: apache-kafka db2 spark-streaming slick-3.0 spark-streaming-kafka

我有以下代码:

val rawWeatherStream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topics)
val parsedWeatherStream: DStream[RawWeatherData] = ingestStream(rawWeatherStream)

我接受来自卡夫卡制作人的任何东西。然后将其发送到ingestStream函数,该函数向我返回所有解析为RawWeatherData()对象的对象。它的代码是:

def ingestStream(rawWeatherStream: InputDStream[(String, String)]): DStream[RawWeatherData] = {
    val parsedWeatherStream = rawWeatherStream.map(_._2.split(","))
      .map(RawWeatherData(_))
    parsedWeatherStream
}

现在,我想使用平滑的方法将parsedWeatherStream中的所有内容保存到DB2数据库中。我的架构是:

case class RawWeatherData(tag: Tag) extends Table[(String, Int, Int, Int, Int, Double, Double, Double, Int, Double, Int, String, Double, Double)](tag , "RawWeatherData") with WeatherModel{
    var wsid = column[String]("wsid")
    var year = column[Int]("year")
    var month = column[Int]("month")
    var day = column[Int]("day")
    var hour = column[Int]("hour")
    var temperature = column[Double]("temperature")
    var dewpoint = column[Double]("depoint")
    var pressure = column[Double]("pressure")
    var windDirection = column[Int]("windDirection")
    var windSpeed = column[Double]("windSpeed")
    var skyCondition = column[Int]("skyCondition")
    var skyConditionText = column[String]("skyConditionText")
    var oneHourPrecip = column[Double]("oneHourPrecip")
    var sixHourPrecip = column[Double]("sixHourPrecip")

    def * = (wsid , year , month , day , hour , temperature , dewpoint , pressure , windDirection , windSpeed , skyCondition , skyConditionText , oneHourPrecip , sixHourPrecip)
}

要将数据保存到数据库中,我编写了以下代码:

val rawWeatherData = TableQuery[com.killrweather.data.RWD]
val db2 = Database.forConfig("db2")
val setup = DBIO.seq(
    // Create the tables, including primary and foreign keys
    rawWeatherData.schema.create,

    // Insert some suppliers
    parsedWeatherStream.foreachRDD { rawWeatherData += _._}
)
//parsedWeatherStream.foreachRDD { rawWeatherData += _._}
val setupFuture = db2.run(setup)

parsedWeatherStream.foreachRDD { rawWeatherData += _._}部分代码是我尝试通过解析所有对象添加到数据库的地方,但是不幸的是,在该行代码中,出现以下错误Type mismatch, found: Unit, required: DBIOAction[_, NoStream, NotInferredE]

那么我现在如何使用Slick将来自Spark Streaming的数据插入数据库?

0 个答案:

没有答案