抛出异常时,spark应用程序以SUCCESS状态完成

时间:2017-12-01 07:36:49

标签: apache-spark yarn

我在纱线上运行一个火花应用程序,我的目标是从jdbc到elasticsearch进行一些ETL 但是,当我检查日志时,会出现一些错误,这个错误是由于网络问题引起的:

17/12/01 00:35:19 WARN scheduler.TaskSetManager: Lost task 1317.0 in stage 0.0 (TID 1381, worker50.hadoop, executor 1): org.apache.spark.util.TaskCompletionListenerException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[192.168.200.154:8201, 192.168.200.156:9200, 192.168.200.155:8201]] 
at org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:138)
at org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:116)
at org.apache.spark.scheduler.Task.run(Task.scala:124)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

这意味着连接失败并在此过程中丢失了一些数据。作业finalStatus应该失败,但是火花通过{"state":"FINISHED","finalStatus":"SUCCEEDED"}返回给我 为什么?我的火花版本是2.2.0

0 个答案:

没有答案