群集环境中EFS文件系统的检查点问题

时间:2018-12-10 08:59:31

标签: apache-spark

在Spark集群环境中将检查点存储到EFS时遇到问题:

  

18/12/09 13:22:33错误CheckpointWriter:无法提交检查点   线程池执行者的任务   java.util.concurrent.RejectedExecutionException:任务   org.apache.spark.streaming.CheckpointWriter$CheckpointWriteHandler@4ed8ebc2   被拒绝   java.util.concurrent.ThreadPoolExecutor@252d7b55 [正在运行,池大小=   1,活动线程= 1,排队的任务= 1000,已完成的任务= 128]   java.util.concurrent.ThreadPoolExecutor $ AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)   在   java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)   在   java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)   在   org.apache.spark.streaming.CheckpointWriter.write(Checkpoint.scala:290)   在   org.apache.spark.streaming.scheduler.JobGenerator.doCheckpoint(JobGenerator.scala:297)   在   org.apache.spark.streaming.scheduler.JobGenerator.org $ apache $ spark $ streaming $ scheduler $ JobGenerator $ processEvent(JobGenerator.scala:186)   在   org.apache.spark.streaming.scheduler.JobGenerator $ anon $ 1.onReceive(JobGenerator.scala:89)   在   org.apache.spark.streaming.scheduler.JobGenerator $ anon $ 1.onReceive(JobGenerator.scala:88)   在org.apache.spark.util.EventLoop $ anon $ 1.run(EventLoop.scala:48)   18/12/09 13:22:33错误CheckpointWriter:无法提交检查点   线程池执行者的任务   java.util.concurrent.RejectedExecutionException:任务   org.apache.spark.streaming.CheckpointWriter$CheckpointWriteHandler@44961d71   被拒绝   java.util.concurrent.ThreadPoolExecutor@252d7b55 [正在运行,池大小=   1,活动线程= 1,排队的任务= 1000,已完成的任务= 128]   java.util.concurrent.ThreadPoolExecutor $ AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)   在   java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)   在   java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)   在   org.apache.spark.streaming.CheckpointWriter.write(Checkpoint.scala:290)   在   org.apache.spark.streaming.scheduler.JobGenerator.doCheckpoint(JobGenerator.scala:297)   在   org.apache.spark.streaming.scheduler.JobGenerator.org $ apache $ spark $ streaming $ scheduler $ JobGenerator $ processEvent(JobGenerator.scala:186)   在   org.apache.spark.streaming.scheduler.JobGenerator $ anon $ 1.onReceive(JobGenerator.scala:89)   在   org.apache.spark.streaming.scheduler.JobGenerator $ anon $ 1.onReceive(JobGenerator.scala:88)   在org.apache.spark.util.EventLoop $ anon $ 1.run(EventLoop.scala:48)处可以

     

任何人都可以帮助我解决该问题。预先感谢。

0 个答案:

没有答案