使用Livy为Zeppelin和HDInsights Spark导入依赖项

时间:2018-08-03 14:00:00

标签: azure apache-spark apache-zeppelin hdinsight livy

我正在尝试编写一个HDInsight Spark应用程序,该应用程序从Azure EventHub读取流数据。我正在将Zeppelin笔记本与Livy解释器一起使用。

我需要导入依赖项

com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.2

为此,我将其添加到

livy.spark.jars.packages

Lip解释器的属性。但是,这破坏了我的代码。即使没有线路

导入org.apache.spark.eventhubs。_

我仍然失败。 (我通常不使用通配符导入,但这只是概念验证应用)

我得到的错误是

org.apache.zeppelin.livy.LivyException: Session 8 is finished, appId: application_[NUMBER], log: [   ApplicationMaster RPC port: -1,     queue: default,     start time: 1533304077387,      final status: UNDEFINED,    tracking URL: http://[LIVY_SERVER_HOSTNAME]:8088/proxy/application_[NUMBER]/,   user: livy, 18/08/03 13:47:57 INFO ShutdownHookManager: Shutdown hook called, 18/08/03 13:47:57 INFO ShutdownHookManager: Deleting directory /tmp/spark-[id], 
YARN Diagnostics: , Application killed by user.]
at org.apache.zeppelin.livy.BaseLivyInterpreter.createSession(BaseLivyInterpreter.java:300)
at org.apache.zeppelin.livy.BaseLivyInterpreter.initLivySession(BaseLivyInterpreter.java:184)
at org.apache.zeppelin.livy.LivySharedInterpreter.open(LivySharedInterpreter.java:57)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at org.apache.zeppelin.livy.BaseLivyInterpreter.getLivySharedInterpreter(BaseLivyInterpreter.java:165)
at org.apache.zeppelin.livy.BaseLivyInterpreter.open(BaseLivyInterpreter.java:139)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:493)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

我怀疑这对Livy或Zeppelin来说确实不是问题,只是我错误地设置了某些配置,或者我需要更改默认设置,可能与下载jar有关。

任何帮助将不胜感激

0 个答案:

没有答案