Question

我正在尝试在YARN上设置spark job服务器我在做

./bin/server_package.sh ec2

这会在/tmp/jobserver中创建一个文件夹当我尝试./sever_start.sh

时

我收到此错误

Uncaught error from thread [JobServer-akka.actor.default-dispatcher-2] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[JobServer]
Uncaught error from thread [JobServer-akka.actor.default-dispatcher-3] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[JobServer]
java.lang.NoClassDefFoundError: ooyala/common/akka/ActorMetrics$$anonfun$receive$1
        at ooyala.common.akka.ActorMetrics$class.receive(ActorMetrics.scala:20)
        at ooyala.common.akka.InstrumentedActor.receive(InstrumentedActor.scala:8)
        at akka.actor.ActorCell.newActor(ActorCell.scala:558)
        at akka.actor.ActorCell.create(ActorCell.scala:578)
        at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
        at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
        at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
        at akka.dispatch.Mailbox.run(Mailbox.scala:219)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Uncaught error from thread [JobServer-akka.actor.default-dispatcher-5] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[JobServer]
java.lang.NoClassDefFoundError: ooyala/common/akka/ActorMetrics$$anonfun$receive$1
        at ooyala.common.akka.ActorMetrics$class.receive(ActorMetrics.scala:20)
        at ooyala.common.akka.InstrumentedActor.receive(InstrumentedActor.scala:8)
        at akka.actor.ActorCell.newActor(ActorCell.scala:558)
        at akka.actor.ActorCell.create(ActorCell.scala:578)
        at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
        at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
        at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
        at akka.dispatch.Mailbox.run(Mailbox.scala:219)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
java.lang.NoClassDefFoundError: scala/runtime/AbstractPartialFunction$mcVL$sp
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

尝试减少版本的akka无效有什么建议吗？

Answer 1

问题似乎是使用scala的冲突版本。 Spark是2.11.7，而作业服务器是2.10.4

Answer 2

您是否使用相同版本的sbt构建与构建spark jobserver的版本？

Answer 3

始终尝试匹配Spark和spark job-server的scala版本。在ec2.sh/local.sh中的版本应该与您在spark-shell中看到的scala版本相同。

在YARN上设置Spark jobserver

3 个答案: