Question

我试图在独立模式下为Spark 1.5.2设置HA（通过zookeeper）。当启动spark时，我得到了这个例外。看起来像是一个不兼容的问题。但是，我不确定如何继续。

15/12/14 10:42:27 ERROR ErrorMonitor: Uncaught fatal error from thread [sparkMaster-akka.actor.default-dispatcher-3] shutting down ActorSystem [sparkMaster]
java.lang.NoSuchMethodError: org.apache.curator.utils.PathUtils.validatePath(Ljava/lang/String;)Ljava/lang/String;
        at org.apache.curator.framework.recipes.leader.LeaderLatch.<init>(LeaderLatch.java:147)
        at org.apache.curator.framework.recipes.leader.LeaderLatch.<init>(LeaderLatch.java:125)
        at org.apache.spark.deploy.master.ZooKeeperLeaderElectionAgent.start(ZooKeeperLeaderElectionAgent.scala:39)
        at org.apache.spark.deploy.master.ZooKeeperLeaderElectionAgent.<init>(ZooKeeperLeaderElectionAgent.scala:34)
        at org.apache.spark.deploy.master.ZooKeeperRecoveryModeFactory.createLeaderElectionAgent(RecoveryModeFactory.scala:74)
        at org.apache.spark.deploy.master.Master.onStart(Master.scala:165)
        at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1$$anonfun$preStart$1.apply$mcV$sp(AkkaRpcEnv.scala:100)
        at org.apache.spark.rpc.akka.AkkaRpcEnv.org$apache$spark$rpc$akka$AkkaRpcEnv$$safelyCall(AkkaRpcEnv.scala:197)
        at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1.preStart(AkkaRpcEnv.scala:99)
        at akka.actor.Actor$class.aroundPreStart(Actor.scala:472)
        at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1.aroundPreStart(AkkaRpcEnv.scala:92)
        at akka.actor.ActorCell.create(ActorCell.scala:580)
        at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
        at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
        at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
        at akka.dispatch.Mailbox.run(Mailbox.scala:219)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
15/12/14 10:42:27 ERROR ActorSystemImpl: Uncaught fatal error from thread [sparkMaster-akka.actor.default-dispatcher-3] shutting down ActorSystem [sparkMaster]
java.lang.NoSuchMethodError: org.apache.curator.utils.PathUtils.validatePath(Ljava/lang/String;)Ljava/lang/String;
        at org.apache.curator.framework.recipes.leader.LeaderLatch.<init>(LeaderLatch.java:147)
        at org.apache.curator.framework.recipes.leader.LeaderLatch.<init>(LeaderLatch.java:125)
        at org.apache.spark.deploy.master.ZooKeeperLeaderElectionAgent.start(ZooKeeperLeaderElectionAgent.scala:39)
        at org.apache.spark.deploy.master.ZooKeeperLeaderElectionAgent.<init>(ZooKeeperLeaderElectionAgent.scala:34)
        at org.apache.spark.deploy.master.ZooKeeperRecoveryModeFactory.createLeaderElectionAgent(RecoveryModeFactory.scala:74)
        at org.apache.spark.deploy.master.Master.onStart(Master.scala:165)
        at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1$$anonfun$preStart$1.apply$mcV$sp(AkkaRpcEnv.scala:100)
        at org.apache.spark.rpc.akka.AkkaRpcEnv.org$apache$spark$rpc$akka$AkkaRpcEnv$$safelyCall(AkkaRpcEnv.scala:197)
        at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1.preStart(AkkaRpcEnv.scala:99)
        at akka.actor.Actor$class.aroundPreStart(Actor.scala:472)
        at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1.aroundPreStart(AkkaRpcEnv.scala:92)
        at akka.actor.ActorCell.create(ActorCell.scala:580)
        at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
        at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
        at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
        at akka.dispatch.Mailbox.run(Mailbox.scala:219)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Answer 1

使用与spark相同的策展人版本

 <dependency>
            <groupId>org.apache.curator</groupId>
            <artifactId>curator-client</artifactId>
            <version>1.5.2</version>
        </dependency>

Answer 2

我在尝试使用 Hadoop 3.3.0 设置高可用性 Spark 2.4.7 时遇到此错误。将 Hadoop 版本降级到 3.2.2 解决了这个问题。

HA中的Spark：java.lang.NoSuchMethodError：org.apache.curator.utils.PathUtils.validatePath（Ljava / lang / String;）Ljava / lang / String;

2 个答案: