Question

我试图通过Windows上的spark提交我的spark-mongo代码jar。我在独立模式下使用spark。我在同一台机器上配置了spark master和两个worker。我想用一个主人和两个工人执行我的jar。我正在尝试执行以下命令：spark-submit --master spark://localhost:7077 --deploy-mode cluster --executor-memory 5G --class spark.mongohadoop.testing3 G:\sparkmon1.jar

我正面临以下错误：

Running Spark using the REST application submission protocol.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/02/28 17:09:13 INFO RestSubmissionClient: Submitting a request to launch an application in spark://192.168.242.1:7077.
17/02/28 17:09:24 WARN RestSubmissionClient: Unable to connect to server spark://192.168.242.1:7077.
Warning: Master endpoint spark://192.168.242.1:7077 was not a REST server. Falling back to legacy submission gateway instead.
17/02/28 17:09:25 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/02/28 17:09:32 ERROR ClientEndpoint: Exception from cluster was: java.lang.NullPointerException
java.lang.NullPointerException
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:482)
        at org.apache.hadoop.util.Shell.run(Shell.java:455)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
        at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:873)
        at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:853)
        at org.apache.spark.util.Utils$.fetchFile(Utils.scala:474)
        at org.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:154)
        at org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:83

我已经在env中设置了winutil路径。为什么我收到这个错误，解决方案是什么？

Answer 1

我在Linux上遇到了同样的错误，但是当我从群集中的特定计算机启动驱动程序时，如果启动驱动程序的请求发送到集群中的任何其他计算机，则表示它正常运行。因此，我的套装似乎是一个环境问题。然后我检查了org.apache.hadoop.util.Shell $ ShellCommandExecutor类中的代码并得到它正在尝试运行命令但在此之前它尝试运行＆＃34; bash＆＃34;为那台机器。我观察到我的bash响应缓慢。在bashrc中做了一些更改并重新启动了我的集群。现在工作正常。

spark submit java.lang.NullPointerException错误

1 个答案: