在apache zeppelin中运行简单的spark任务时出现异常

时间：2017-02-22 09:25:03

标签： apache-spark apache-zeppelin

我在Apache Zeppelin中尝试tu run simple spark命令时收到了InvocationTargetException。下面你可以看到两个尝试运行的任务。两者都给了我同样的例外。

enter image description here 来自Zeppelin的Stacktrace：

DEBUG [2017-02-22 10:14:34,351] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) - ERROR [2017-02-22 10:14:34,297] ({pool-2-thread-5} Utils.java[invokeMethod]:40) -
DEBUG [2017-02-22 10:14:34,353] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) - java.lang.reflect.InvocationTargetException
DEBUG [2017-02-22 10:14:34,354] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
DEBUG [2017-02-22 10:14:34,356] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
DEBUG [2017-02-22 10:14:34,357] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
DEBUG [2017-02-22 10:14:34,358] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.lang.reflect.Method.invoke(Method.java:498)
DEBUG [2017-02-22 10:14:34,361] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:38)
DEBUG [2017-02-22 10:14:34,361] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:33)
DEBUG [2017-02-22 10:14:34,365] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.spark.SparkInterpreter.createSparkSession(SparkInterpreter.java:350)
DEBUG [2017-02-22 10:14:34,368] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.spark.SparkInterpreter.getSparkSession(SparkInterpreter.java:222)
DEBUG [2017-02-22 10:14:34,368] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:815)
DEBUG [2017-02-22 10:14:34,371] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
DEBUG [2017-02-22 10:14:34,371] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:482)
DEBUG [2017-02-22 10:14:34,375] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
DEBUG [2017-02-22 10:14:34,375] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
DEBUG [2017-02-22 10:14:34,377] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
DEBUG [2017-02-22 10:14:34,378] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
DEBUG [2017-02-22 10:14:34,380] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
DEBUG [2017-02-22 10:14:34,381] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
DEBUG [2017-02-22 10:14:34,381] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
DEBUG [2017-02-22 10:14:34,385] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
DEBUG [2017-02-22 10:14:34,386] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.lang.Thread.run(Thread.java:745)
DEBUG [2017-02-22 10:14:34,387] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) - Caused by: java.lang.NullPointerException
DEBUG [2017-02-22 10:14:34,387] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
DEBUG [2017-02-22 10:14:34,389] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.hadoop.util.Shell.runCommand(Shell.java:482)
DEBUG [2017-02-22 10:14:34,389] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.hadoop.util.Shell.run(Shell.java:455)
DEBUG [2017-02-22 10:14:34,389] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
DEBUG [2017-02-22 10:14:34,391] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:873)
DEBUG [2017-02-22 10:14:34,392] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:853)
DEBUG [2017-02-22 10:14:34,394] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.spark.util.Utils$.fetchFile(Utils.scala:475)
DEBUG [2017-02-22 10:14:34,394] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.spark.SparkContext.addFile(SparkContext.scala:1477)
DEBUG [2017-02-22 10:14:34,395] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.spark.SparkContext.addFile(SparkContext.scala:1425)
DEBUG [2017-02-22 10:14:34,397] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.spark.SparkContext$$anonfun$12.apply(SparkContext.scala:470)
DEBUG [2017-02-22 10:14:34,398] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.spark.SparkContext$$anonfun$12.apply(SparkContext.scala:470)
DEBUG [2017-02-22 10:14:34,400] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at scala.collection.immutable.List.foreach(List.scala:381)
DEBUG [2017-02-22 10:14:34,401] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.spark.SparkContext.<init>(SparkContext.scala:470)
DEBUG [2017-02-22 10:14:34,401] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
DEBUG [2017-02-22 10:14:34,403] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
 INFO [2017-02-22 10:14:34,403] ({pool-2-thread-3} SchedulerFactory.java[jobFinished]:137) - Job paragraph_1487752190357_-397170656 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session969841713
DEBUG [2017-02-22 10:14:34,404] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
DEBUG [2017-02-22 10:14:34,407] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at scala.Option.getOrElse(Option.scala:121)
DEBUG [2017-02-22 10:14:34,407] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
DEBUG [2017-02-22 10:14:34,409] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  ... 20 more
DEBUG [2017-02-22 10:14:34,409] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  INFO [2017-02-22 10:14:34,319] ({pool-2-thread-5} SparkInterpreter.java[createSparkSession]:351) - Created Spark session with Hive support
DEBUG [2017-02-22 10:14:34,410] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) - ERROR [2017-02-22 10:14:34,319] ({pool-2-thread-5} Job.java[run]:181) - Job failed
DEBUG [2017-02-22 10:14:34,412] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) - java.lang.NullPointerException
DEBUG [2017-02-22 10:14:34,412] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:38)
DEBUG [2017-02-22 10:14:34,412] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:33)
DEBUG [2017-02-22 10:14:34,414] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.spark.SparkInterpreter.createSparkContext_2(SparkInterpreter.java:380)
DEBUG [2017-02-22 10:14:34,415] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:369)
DEBUG [2017-02-22 10:14:34,415] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:144)
DEBUG [2017-02-22 10:14:34,417] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:817)
DEBUG [2017-02-22 10:14:34,418] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
DEBUG [2017-02-22 10:14:34,420] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:482)
DEBUG [2017-02-22 10:14:34,423] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
DEBUG [2017-02-22 10:14:34,423] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
DEBUG [2017-02-22 10:14:34,425] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
DEBUG [2017-02-22 10:14:34,425] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
DEBUG [2017-02-22 10:14:34,427] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
DEBUG [2017-02-22 10:14:34,428] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
DEBUG [2017-02-22 10:14:34,428] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
DEBUG [2017-02-22 10:14:34,430] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
DEBUG [2017-02-22 10:14:34,431] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.lang.Thread.run(Thread.java:745)
DEBUG [2017-02-22 10:14:34,431] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  INFO [2017-02-22 10:14:34,320] ({pool-2-thread-5} SchedulerFactory.java[jobFinished]:137) - Job remoteInterpretJob_1487754873203 finished by scheduler org.apache.zeppelin.spark.SparkInterpreter1927937575
DEBUG [2017-02-22 10:14:34,433] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) - DEBUG [2017-02-22 10:14:34,321] ({pool-1-thread-3} Interpreter.java[getProperty]:165) - key: zeppelin.spark.concurrentSQL, value: false
DEBUG [2017-02-22 10:14:34,433] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) - DEBUG [2017-02-22 10:14:34,321] ({pool-1-thread-3} Interpreter.java[getProperty]:165) - key: zeppelin.spark.concurrentSQL, value: false

我使用Spark＆＃34; spark-2.1.0-bin-hadoop2.7＆＃34;，hadoop＆＃34; hadoop-2.7.3＆＃34;和zeppelin＆＃34; zeppelin-0.7.0-bin-all＆＃34;。

1 个答案:

答案 0 :(得分：0)

你必须为hadoop 2.7构建zeppelin

以下是built from source

的文档 
# make sure that you are using scala 2.11.x, maven 3.1.x+ and JDK 1.7+
# also, you need nodejs 6.9.1+ (will be removed from build requirements in 0.7.1)

$ ./dev/change_scala_version.sh 2.11
$ mvn clean package -DskipTests -Pspark-2.1 -Phadoop-2.7 -Pyarn -Ppyspark -Psparkr -Pr -Pscala-2.11
相关问题
最新问题