我正在尝试编写一个HDInsight Spark应用程序,该应用程序从Azure EventHub读取流数据。我正在将Zeppelin笔记本与Livy解释器一起使用。
我需要导入依赖项
com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.2
为此,我将其添加到
livy.spark.jars.packages
Lip解释器的属性。但是,这破坏了我的代码。即使没有线路导入org.apache.spark.eventhubs。_
我仍然失败。 (我通常不使用通配符导入,但这只是概念验证应用)
我得到的错误是
org.apache.zeppelin.livy.LivyException: Session 8 is finished, appId: application_[NUMBER], log: [ ApplicationMaster RPC port: -1, queue: default, start time: 1533304077387, final status: UNDEFINED, tracking URL: http://[LIVY_SERVER_HOSTNAME]:8088/proxy/application_[NUMBER]/, user: livy, 18/08/03 13:47:57 INFO ShutdownHookManager: Shutdown hook called, 18/08/03 13:47:57 INFO ShutdownHookManager: Deleting directory /tmp/spark-[id],
YARN Diagnostics: , Application killed by user.]
at org.apache.zeppelin.livy.BaseLivyInterpreter.createSession(BaseLivyInterpreter.java:300)
at org.apache.zeppelin.livy.BaseLivyInterpreter.initLivySession(BaseLivyInterpreter.java:184)
at org.apache.zeppelin.livy.LivySharedInterpreter.open(LivySharedInterpreter.java:57)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at org.apache.zeppelin.livy.BaseLivyInterpreter.getLivySharedInterpreter(BaseLivyInterpreter.java:165)
at org.apache.zeppelin.livy.BaseLivyInterpreter.open(BaseLivyInterpreter.java:139)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:493)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
我怀疑这对Livy或Zeppelin来说确实不是问题,只是我错误地设置了某些配置,或者我需要更改默认设置,可能与下载jar有关。
任何帮助将不胜感激