Question

我一直在尝试使用h2o（master=yarn）运行Spark 2.2，rsparkling但是当我运行h2o_context(sc)时，我会遇到异常：

Error: java.lang.NoSuchMethodError: org.apache.spark.util.Utils$.getUserJars(Lorg/apache/spark/SparkConf;Z)Lscala/collection/Seq;
    at org.apache.spark.repl.h2o.H2OInterpreter.createSettings(H2OInterpreter.scala:66)
    at org.apache.spark.repl.h2o.BaseH2OInterpreter.initializeInterpreter(BaseH2OInterpreter.scala:101)
    at org.apache.spark.repl.h2o.BaseH2OInterpreter.<init>(BaseH2OInterpreter.scala:291)
    at org.apache.spark.repl.h2o.H2OInterpreter.<init>(H2OInterpreter.scala:42)
    at water.api.scalaInt.ScalaCodeHandler.createInterpreterInPool(ScalaCodeHandler.scala:100)
    at water.api.scalaInt.ScalaCodeHandler$$anonfun$initializeInterpreterPool$1.apply(ScalaCodeHandler.scala:94)
    at water.api.scalaInt.ScalaCodeHandler$$anonfun$initializeInterpreterPool$1.apply(ScalaCodeHandler.scala:93)
    at scala.collection.immutable.Range.foreach(Range.scala:160)
    at water.api.scalaInt.ScalaCodeHandler.initializeInterpreterPool(ScalaCodeHandler.scala:93)
    at water.api.scalaInt.ScalaCodeHandler.<init>(ScalaCodeHandler.scala:37)
    at water.api.scalaInt.ScalaCodeHandler$.registerEndpoints(ScalaCodeHandler.scala:132)
    at water.api.CoreRestAPI$.registerEndpoints(CoreRestAPI.scala:32)
    at water.api.RestAPIManager.register(RestAPIManager.scala:39)
    at water.api.RestAPIManager.registerAll(RestAPIManager.scala:31)
    at org.apache.spark.h2o.backends.internal.InternalH2OBackend.init(InternalH2OBackend.scala:117)
    at org.apache.spark.h2o.H2OContext.init(H2OContext.scala:121)
    at org.apache.spark.h2o.H2OContext$.getOrCreate(H2OContext.scala:352)
    at org.apache.spark.h2o.H2OContext$.getOrCreate(H2OContext.scala:387)
    at org.apache.spark.h2o.H2OContext.getOrCreate(H2OContext.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at sparklyr.Invoke$.invoke(invoke.scala:102)
    at sparklyr.StreamHandler$.handleMethodCall(stream.scala:97)
    at sparklyr.StreamHandler$.read(stream.scala:62)
    at sparklyr.BackendHandler.channelRead0(handler.scala:52)
    at sparklyr.BackendHandler.channelRead0(handler.scala:14)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:643)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:566)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
    at java.lang.Thread.run(Thread.java:745)

我也尝试过spark 2.0.0（通过sparklyr函数安装：spark_install）。然后匹配rsparkling和h2o。我设置master="local"时有效，但是当我设置master="yarn"

时，它会出现同样的错误

同样地，我尝试了Spark 1.6，它运行良好（也master=yarn）。

有什么想法吗？

这是我的代码：

library(sparklyr)
library(rsparkling)
library(h2o)


Sys.setenv(SPARK_HOME='/usr/hdp/2.6.3.0-235/spark2')

sc <- spark_connect(master = "yarn")
h2o_context(sc)

我尝试使用（类似的）install.packages（“h2o”，type =“source”，repos =“http://h2o-release.s3.amazonaws.com/h2o/rel-tverberg/2/R”）安装各种不同版本的h2o，但每次错误都没有变化

Answer 1

感谢错误报告！它应该已经在Sparkling Water的主分支中修复，它将成为下一个版本的一部分（ETA是下周）。

谢谢，纳瓦迪普

运行h2o，rsparkling，sparklyr

1 个答案: