我正在编写一个测试应用程序,该应用程序使用来自Kafka的topcis的消息,然后将数据推送到S3并进入RDBMS表(流程类似于此处所示:https://databricks.com/blog/2017/04/26/processing-data-in-apache-kafka-with-structured-streaming-in-apache-spark-2-2.html)。所以我从Kafka读取数据然后:
所以我喜欢:
Dataset<Row> df = spark
.readStream()
.format("kafka")
.option("kafka.bootstrap.servers", "host1:port1,host2:port2")
.option("subscribe", "topic1,topic2,topic3")
.option("startingOffsets", "earliest")
.load()
.select(from_json(col("value").cast("string"), schema, jsonOptions).alias("parsed_value"))
(请注意我从多个Kafka主题中读到的内容)。 接下来,我定义了所需的数据集:
Dataset<Row> allMessages = df.select(.....)
Dataset<Row> messagesOfType1 = df.select() //some unique conditions applied on JSON elements
Dataset<Row> messagesOfType2 = df.select() //some other unique conditions
现在为每个数据集我创建查询以开始处理:
StreamingQuery s3Query = allMessages
.writeStream()
.format("parquet")
.option("startingOffsets", "latest")
.option("path", "s3_location")
.start()
StreamingQuery firstQuery = messagesOfType1
.writeStream()
.foreach(new CustomForEachWiriterType1()) // class that extends ForeachWriter[T] and save data into external RDBMS table
.start();
StreamingQuery secondQuery = messagesOfType2
.writeStream()
.foreach(new CustomForEachWiriterType2()) // class that extends ForeachWriter[T] and save data into external RDBMS table (may be even another database than before)
.start();
现在我想知道:
那些并行执行的查询(或者以FIFO顺序一个接一个地执行,我应该将这些查询分配给不同的调度程序池)?
答案 0 :(得分:7)
将是那些并行执行的查询
是。这些查询将并行执行(每个trigger
您没有指定,因此要尽快运行它们。)
在内部,当您在DataStreamWriter上执行start
时,您会创建一个StreamExecution
,然后立即创建所谓的守护程序microBatchThread
(引自the Spark source code下面):
val microBatchThread =
new StreamExecutionThread(s"stream execution thread for $prettyIdString") {
override def run(): Unit = {
// To fix call site like "run at <unknown>:0", we bridge the call site from the caller
// thread to this micro batch thread
sparkSession.sparkContext.setCallSite(callSite)
runBatches()
}
}
您可以在其自己的线程中查看名称为
的每个查询stream execution thread for [prettyIdString]