使用crontab运行Spark作业

时间:2020-10-11 09:24:06

标签: apache-spark hadoop cron

我已经编写了shell脚本来运行spark作业:

/home/ajit/deviceAnalysis.sh
/usr/spark/bin/spark-submit --master yarn --deploy-mode cluster --packages org.mongodb.spark:mongo-spark-connector_2.11:2.4.1 --conf spark.yarn.maxAppAttempts=4 --conf spark.yarn.am.attemptFailuresValidityInterval=1h --class com.mmi.probes.analysis.app.StartApp --driver-memory 2g --queue deviceAnalysis ./DailyDeviceProbesAnalysis.jar

({{1)}可以正常工作。但是我想使用crontab每5分钟运行一次,但运行时却抛出异常:

sh /home/ajit/deviceAnalysis.sh

在crontab中 java.io.FileNotFoundException: File file:/home/ajit/.sparkStaging/application_1601983398886_0126/__spark_libs__9052532821082604838.zip does not exist 请帮我现在该怎么办

0 个答案:

没有答案