Question

我正在从kafka播放火花。我想将rdd从kafka转换为dataframe。我正在使用以下方法。 val ssc = new StreamingContext（“ local [*]”，“ KafkaExample”，Seconds（4））

    val kafkaParams = Map[String, Object](
    "bootstrap.servers" -> "dofff2.dl.uk.feefr.com:8002",
    "security.protocol" -> "SASL_PLAINTEXT",
    "key.deserializer" -> classOf[StringDeserializer],
    "value.deserializer" -> classOf[StringDeserializer],
    "group.id" -> "1",
    "auto.offset.reset" -> "latest",
    "enable.auto.commit" -> (false: java.lang.Boolean)
    )

   val topics = Array("csv")
   val stream = KafkaUtils.createDirectStream[String, String](
   ssc,
   PreferConsistent,
   Subscribe[String, String](topics, kafkaParams)
   )

   val strmk = stream.map(record => (record.value))
  val rdd1 = strmk.map(line => line.split(',')).map(s => (s(0).toString, s(1).toString,s(2).toString,s(3).toString,s(4).toString, s(5).toString,s(6).toString,s(7).toString))

  rdd1.foreachRDD((rdd, time) => {
  val sqlContext = SQLContextSingleton.getInstance(rdd.sparkContext)
  import sqlContext.implicits._
  val requestsDataFrame = rdd.map(w => Record(w._1, w._2, w._3,w._4, w._5, w._6,w._7, w._8)).toDF()
  requestsDataFrame.createOrReplaceTempView("requests")
  val word_df =sqlContext.sql("select * from  requests ")
  println(s"========= $time =========")
  word_df.show()
  })

但是在数据框中，我也想包括来自kafka的时间戳。有人可以帮忙吗？

Answer 1

Kafka记录具有各种属性。

请参见https://spark.apache.org/docs/2.2.0/structured-streaming-kafka-integration.html

请注意，Kafka有流式和批处理方法。

一个例子：

import java.sql.Timestamp

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._
import org.apache.spark.sql.streaming.OutputMode

val sparkSession = SparkSession.builder
  .master("local")
  .appName("example")
  .getOrCreate()

import sparkSession.implicits._
sparkSession.sparkContext.setLogLevel("ERROR")
val socketStreamDs = sparkSession.readStream
  .format("kafka")
  .option("kafka.bootstrap.servers", "localhost:9092")  
  .option("subscribe", "AAA")     
  .option("startingOffsets", "earliest")
  .load()
  //.as[String]
  //
  //.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)", "CAST(timestamp AS STRING)")
  .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)", "timestamp")
  .writeStream
    .format("console")
    .option("truncate", "false")
    .outputMode(OutputMode.Append())
    .start().awaitTermination()

我的示例输出如下：

-------------------------------------------
Batch: 0
-------------------------------------------
+----+-----+-----------------------+
|key |value|timestamp              |
+----+-----+-----------------------+
|null|RRR  |2019-02-07 04:37:34.983|
|null|HHH  |2019-02-07 04:37:36.802|
|null|JJJ  |2019-02-07 04:37:39.1  |
+----+-----+-----------------------+

对于非结构化流，

您只需要在上面展开声明即可

stream.map { record => (record.timestamp(), record.key(), record.value()) }

如何在转换为DF的过程中从kafka添加时间戳以触发流式传输

1 个答案: