从spark-submit

时间:2018-06-06 15:40:07

标签: apache-spark pyspark

无论如何,我可以捕获火花提交作业的输出

spark-submit --master yarn --deploy-mode cluster --queue root.dm.xxx --principal $USER@xxx.CO.UK --keytab~ / $ USER.keytab --driver-class-路径" / opt / cloudera / parcels / CDH / lib / hive / lib / " --driver-java-options" -Dspark.executor.extraClassPath = / opt / cloudera / parcels / CDH / lib / hive / lib / " --conf" spark.yarn.appMasterEnv.PYSPARK_PYTHON = / dhcommon / dhpython / python / bin / python2.7" --num-executors 12 --executor-memory 15GB --executor-cores 5 --conf spark.yarn.executor.memoryOverhead = 3072 --conf" spark.yarn.appMasterEnv.PYSPARK_DRIVER_PYTHON = / dhcommon / dhpython /蟒/ bin中/ python2.7" --py-files" /home/xxx/eggs/xxx-1.0-py2.7.egg,/home/xxx/eggs/xxx-1.0-py2.7.egg" simple.py

1 个答案:

答案 0 :(得分:1)

shell变量$?具有最后一个命令的返回值。你可以这样做:

spark-submit --master yarn ....
ret_val=$?  # In case you want to reference the return code more than once
...

这当然是一种方法。您可以通过其他方式查看此blog