Apache Spark错误的akka​​-remote netty版本

时间：2015-11-06 18:47:24

标签： apache-spark akka netty akka-remote-actor

当火花正在运行测试与sbt。我得到了这个例外：

18:58:49.049 [sparkDriver-akka.actor.default-dispatcher-2] ERROR akka.actor.ActorSystemImpl - Uncaught fatal error from thread [sparkDriver-akka.remote.default-remote-dispatcher-5] shutting down ActorSystem [sparkDriver]
java.lang.VerifyError: (class: org/jboss/netty/channel/socket/nio/NioWorkerPool, method: createWorker signature: (Ljava/util/concurrent/Executor;)Lorg/jboss/netty/channel/socket/nio/AbstractNioWorker;) Wrong return type in function
at akka.remote.transport.netty.NettyTransport.<init>(NettyTransport.scala:283) ~[akka-remote_2.11-2.3.4-spark.jar:na]
at akka.remote.transport.netty.NettyTransport.<init>(NettyTransport.scala:240) ~[akka-remote_2.11-2.3.4-spark.jar:na]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_45]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_45]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_45]
at java.lang.reflect.Constructor.newInstance(Constructor.java:422) ~[na:1.8.0_45]
at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(DynamicAccess.scala:78) ~[akka-actor_2.11-2.3.4-spark.jar:na]
at scala.util.Try$.apply(Try.scala:192) ~[scala-library-2.11.7.jar:0.13.8]
at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:73) ~[akka-actor_2.11-2.3.4-spark.jar:na]
at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84) ~[akka-actor_2.11-2.3.4-spark.jar:na]
at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84) ~[akka-actor_2.11-2.3.4-spark.jar:na]
at scala.util.Success.flatMap(Try.scala:231) ~[scala-library-2.11.7.jar:0.13.8]
at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:84) ~[akka-actor_2.11-2.3.4-spark.jar:na]
at akka.remote.EndpointManager$$anonfun$9.apply(Remoting.scala:692) ~[akka-remote_2.11-2.3.4-spark.jar:na]
at akka.remote.EndpointManager$$anonfun$9.apply(Remoting.scala:684) ~[akka-remote_2.11-2.3.4-spark.jar:na]
at scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:728) ~[scala-library-2.11.7.jar:0.13.8]
at scala.collection.Iterator$class.foreach(Iterator.scala:742) ~[scala-library-2.11.7.jar:0.13.8]
at scala.collection.AbstractIterator.foreach(Iterator.scala:1194) ~[scala-library-2.11.7.jar:0.13.8]
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) ~[scala-library-2.11.7.jar:0.13.8]
at scala.collection.AbstractIterable.foreach(Iterable.scala:54) ~[scala-library-2.11.7.jar:0.13.8]
at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:727) ~[scala-library-2.11.7.jar:0.13.8]
at akka.remote.EndpointManager.akka$remote$EndpointManager$$listens(Remoting.scala:684) ~[akka-remote_2.11-2.3.4-spark.jar:na]
at akka.remote.EndpointManager$$anonfun$receive$2.applyOrElse(Remoting.scala:492) ~[akka-remote_2.11-2.3.4-spark.jar:na]
at akka.actor.Actor$class.aroundReceive(Actor.scala:465) ~[akka-actor_2.11-2.3.4-spark.jar:na]
at akka.remote.EndpointManager.aroundReceive(Remoting.scala:395) ~[akka-remote_2.11-2.3.4-spark.jar:na]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) [akka-actor_2.11-2.3.4-spark.jar:na]
at akka.actor.ActorCell.invoke(ActorCell.scala:487) [akka-actor_2.11-2.3.4-spark.jar:na]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) [akka-actor_2.11-2.3.4-spark.jar:na]
at akka.dispatch.Mailbox.run(Mailbox.scala:220) [akka-actor_2.11-2.3.4-spark.jar:na]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) [akka-actor_2.11-2.3.4-spark.jar:na]
...

我首先认为它是由我的一个子库导入的错误的netty版本。

在查看我的依赖图后，我发现了4个不同版本的netty。 3.6.6 3.8.0 3.9.3 4.0.23

这些版本大多是由spark自己导入的：o 4.0.23由spark直接导入，3.8.0由其子依赖Akka-remote

导入

我尝试使用excludeAll( ExclusionRule(organization = "io.netty"))从我的所有子依赖项中排除netty依赖项，并逐个添加每个netty版本。但它并没有解决问题。 我还尝试排除所有使用com.typesafe.akka的{​​{1}}个deps我遇到同样的问题。

Akka-remote 2.4.0所需的网络版本是Akka-remote。甚至迫使这个人不起作用。我还强迫我的项目使用空间火花Akka版本3.8.0，它不会改变任何东西。

获取信息我的build.sbt 

Akka-remote:3.8.0-spark

如您所见，我尝试排除尝试的最大限度事项

并且子项目包含

libraryDependencies ++= Seq(
  "com.datastax.cassandra"  %   "cassandra-driver-core"               % "2.1.7.1" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
  "com.datastax.spark"      %%  "spark-cassandra-connector"           % "1.4.0" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
  "com.datastax.spark"      %%  "spark-cassandra-connector-embedded"  % "1.4.0"  % Test excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
  "xxx.xxxxx"               %%  "shed"                                % "0.10.0-MOK-1848-DEBUG-SNAPSHOT" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
  "com.twitter"             %%  "util-collection"                     % "6.27.0" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
  "io.circe"                %%  "circe-core"                          % "0.2.0-SNAPSHOT" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
  "io.circe"                %%  "circe-generic"                       % "0.2.0-SNAPSHOT" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
  "io.circe"                %%  "circe-jawn"                          % "0.2.0-SNAPSHOT" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
  "io.netty"                %   "netty"                               % "3.6.6.Final",
  //"com.typesafe.akka"       %   "akka-remote_2.11"                    % "2.3.4",
  "org.apache.cassandra"    %   "cassandra-all"                       % "2.1.5" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
  "org.apache.cassandra"    %   "cassandra-thrift"                    % "2.0.5" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
  "org.apache.spark"        %%  "spark-streaming-kafka"               % "1.4.0" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
  "org.apache.spark"        %%  "spark-streaming"                     % sparkVersion % "provided" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
  "org.apache.spark"        %%  "spark-core"                          % sparkVersion % "provided" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
  "org.apache.spark"        %%  "spark-sql"                           % sparkVersion % "provided" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
  "org.scalaz.stream"       %   "scalaz-stream_2.11"                  % "0.7.3" % Test excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
  "org.specs2"              %%  "specs2-core"                         % "3.6.1-scalaz-7.0.6" % "test" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka"))
)

1 个答案:

答案 0 :(得分：6)

这是一个陷阱！

netty组织过去发生了变化。 从org.jboss.netty到io.netty，但它们包含相同的包。

排除（＆＃34; org.jboss.netty＆＃34;，＆＃34; netty＆＃34;）解决我的问题。

相关问题
最新问题