处理多个Kafka主题

时间:2018-09-17 06:10:42

标签: apache-spark spark-structured-streaming

我将Spark结构化流媒体与Kafka集成在一起,其中我正在听2个主题

def main(args: Array[String]): Unit = {

    val schema = StructType(
      List(
        StructField("gatewayId", StringType, true),
        StructField("userId", StringType, true)
      )
    )

    val spark = SparkSession
      .builder
      .master("local[4]")
      .appName("DeviceAutomation")
      .getOrCreate()

    val dfStatus = spark.readStream.
      format("kafka").
      option("subscribe", "utility-status").
      option("kafka.bootstrap.servers", "localhost:9092").
      option("startingOffsets", "earliest")
      .load()


      val dfCritical = spark.readStream.
      format("kafka").
      option("subscribe", "utility-critical").
      option("kafka.bootstrap.servers", "localhost:9092").
      option("startingOffsets", "earliest")
      .load()

      }

由于我将列出更多主题并执行不同的操作,因此我希望将每个主题移到单独的类中,以提高清晰度。是否有可能,以及如何从主班级启动它们?

0 个答案:

没有答案