Question

我有一个处理某些事件的Spark Streaming应用程序。

有时，如果获得特定事件，我想停止应用程序。我在驱动程序中收集执行程序的结果，并根据这些结果，使用StreamingContext.stop（stopSparkContext = true）终止StreamingContext。

当我这样做时，我可以在日志中看到应用程序正在关闭，关闭接收器等。

但是当我进入主人的网络用户界面时，我仍然可以在“正在运行的应用程序”下看到该应用程序。但如果我点击它，它表示端点不存在。当我检查机器上的打开进程时，我可以看到作业的进程仍在运行。

我关闭应用程序错了吗？

一旦我调用stop（）方法，那些就是日志：

2018-03-28 11:59:04 INFO  KafkaProducer:615 - Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
2018-03-28 11:59:04 INFO  ReceiverTracker:54 - Sent stop signal to all 1 receivers
2018-03-28 11:59:05 INFO  BlockManagerInfo:54 - Added input-0-1522238344750 in memory on i-va-spark1:59059 (size: 1632.0 B, free: 579.2 MB)
2018-03-28 11:59:05 ERROR ReceiverTracker:70 - Deregistered receiver for stream 0: Stopped by driver
2018-03-28 11:59:05 INFO  BlockManagerInfo:54 - Added input-0-1522238345000 in memory on i-va-spark1:59059 (size: 272.0 B, free: 579.2 MB)
2018-03-28 11:59:05 INFO  TaskSetManager:54 - Finished task 0.0 in stage 2.0 (TID 70) in 30213 ms on i-va-spark1 (executor 0) (1/1)
2018-03-28 11:59:05 INFO  TaskSchedulerImpl:54 - Removed TaskSet 2.0, whose tasks have all completed, from pool
2018-03-28 11:59:05 INFO  DAGScheduler:54 - ResultStage 2 (start at UserLocationHistoryJob.scala:38) finished in 30.213 s
2018-03-28 11:59:05 INFO  ReceiverTracker:54 - All of the receivers have deregistered successfully
2018-03-28 11:59:05 INFO  ReceiverTracker:54 - ReceiverTracker stopped
2018-03-28 11:59:05 INFO  JobGenerator:54 - Stopping JobGenerator immediately
2018-03-28 11:59:05 INFO  RecurringTimer:54 - Stopped timer for JobGenerator after time 1522238340000
2018-03-28 11:59:05 INFO  JobGenerator:54 - Stopped JobGenerator
2018-03-28 11:59:07 INFO  JobScheduler:54 - Stopped JobScheduler
2018-03-28 11:59:07 INFO  StreamingContext:54 - StreamingContext stopped successfully
2018-03-28 11:59:07 INFO  BlockManagerInfo:54 - Removed broadcast_5_piece0 on 10.0.0.243:41976 in memory (size: 2.4 KB, free: 488.4 MB)
2018-03-28 11:59:07 INFO  BlockManagerInfo:54 - Removed broadcast_5_piece0 on i-va-spark1:59059 in memory (size: 2.4 KB, free: 579.2 MB)
2018-03-28 11:59:07 INFO  BlockManagerInfo:54 - Removed broadcast_4_piece0 on 10.0.0.243:41976 in memory (size: 23.9 KB, free: 488.4 MB)
2018-03-28 11:59:07 INFO  BlockManagerInfo:54 - Removed broadcast_4_piece0 on i-va-spark1:59059 in memory (size: 23.9 KB, free: 579.2 MB)
2018-03-28 11:59:37 WARN  QueuedThreadPool:178 - SparkUI{STOPPING,8<=9<=200,i=0,q=4} Couldn't stop Thread[SparkUI-171-selector-ServerConnectorManager@478b3e9/2,5,main]
2018-03-28 11:59:37 WARN  QueuedThreadPool:178 - SparkUI{STOPPING,8<=9<=200,i=0,q=4} Couldn't stop Thread[SparkUI-172-selector-ServerConnectorManager@478b3e9/3,5,main]
2018-03-28 11:59:37 WARN  QueuedThreadPool:178 - SparkUI{STOPPING,8<=9<=200,i=0,q=4} Couldn't stop Thread[SparkUI-169-selector-ServerConnectorManager@478b3e9/0,5,main]
2018-03-28 11:59:37 WARN  QueuedThreadPool:178 - SparkUI{STOPPING,8<=9<=200,i=0,q=4} Couldn't stop Thread[SparkUI-170-selector-ServerConnectorManager@478b3e9/1,5,main]
2018-03-28 13:22:01 INFO  DiskBlockManager:54 - Shutdown hook called
2018-03-28 13:22:01 INFO  ShutdownHookManager:54 - Shutdown hook called
2018-03-28 13:22:01 INFO  ShutdownHookManager:54 - Deleting directory /data/spark/scratch/spark-69a3a8a6-5504-4153-a4c1-059676861581
2018-03-28 13:22:01 INFO  ShutdownHookManager:54 - Deleting directory /data/spark/scratch/spark-69a3a8a6-5504-4153-a4c1-059676861581/userFiles-8a970eec-da41-442b-9ccf-1621b9e9e045

修改关闭应用的代码：

def handleStream(eventsStream: DStream[ETLEvent], appName: Broadcast[String], appId: Broadcast[String])(ssc: StreamingContext, saveTopic: String): Unit = {
    eventsStream.filter(_ != null).foreachRDD { rdd =>
      val recordsAmount = rdd.count()
      val partitionsAmount = Seq(1, (recordsAmount.toDouble / maxPartitionSize).ceil.toInt).max
      flowLogger.info(s"Splitting $recordsAmount record from ${rdd.getNumPartitions} to $partitionsAmount partitions")
      val partitionedRDD = rdd.repartition(partitionsAmount)
      val rddVals = partitionedRDD.map(Decoder.requestFromETLEvent)
      val results = handleRDD(rddVals, appName, appId).collect()
      killAppBatchIfNeeded[MyRequest](results)(ssc, saveTopic, MyExporter)
    }

  }

  def killAppBatchIfNeeded[I](results: Seq[(Seq[I], Try[_])])
                             (ssc: StreamingContext, saveTopic: String, exporter: IResponsesExporter[_])
                             (implicit writes: Writes[I]): Unit = {
    val failedRequests = results.filter(_._2.isFailure).map { case (req, tryResp) => (req, tryResp.failed.get) }
    failedRequests.foreach { case (req, error) =>
      logger.error(s"Request ${Json.toJson(req)} failed.", error)
      req.foreach(exporter.saveRequest(saveTopic, _))
    }
    if (failedRequests.nonEmpty) {
      logger.warn("Some messages had failures, stopping job")
      ssc.stop(true)
    }
  }

Spark Streaming - 以编程方式杀死应用程序

0 个答案: