如何在本地模式下使用spark运行Sparkling Water示例

时间:2017-01-06 03:26:18

标签: h2o sparkling-water

我正在尝试在IntelliJ IDEA中运行苏打水深度学习演示 代码链接是:

https://github.com/h2oai/sparkling-water/blob/RELEASE-2.0.3/examples/src/main/scala/org/apache/spark/examples/h2o/DeepLearningDemo.scala

如果无法启动,则例外是:

17/01/06 11:18:41 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerJobEnd(1,1483672721446,JobFailed(org.apache.spark.SparkException: Job 1 cancelled because SparkContext was shut down))
    at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:187)
    at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:177)
    at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
    at java.lang.Thread.getStackTrace(Thread.java:1108)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:632)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1890)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1903)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1916)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1930)
    at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:912)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:358)
    at org.apache.spark.rdd.RDD.collect(RDD.scala:911)
    at org.apache.spark.h2o.backends.internal.InternalBackendUtils$class.startH2O(InternalBackendUtils.scala:163)
    at org.apache.spark.h2o.backends.internal.InternalBackendUtils$.startH2O(InternalBackendUtils.scala:262)
    at org.apache.spark.h2o.backends.internal.InternalH2OBackend.init(InternalH2OBackend.scala:99)
    at org.apache.spark.h2o.H2OContext.init(H2OContext.scala:102)
    at org.apache.spark.h2o.H2OContext$.getOrCreate(H2OContext.scala:279)
    at org.apache.spark.h2o.H2OContext$.getOrCreate(H2OContext.scala:301)
    at com.xyz.HelloSparklingWater$.main(HelloSparklingWater.scala:35)
    at com.xyz.HelloSparklingWater.main(HelloSparklingWater.scala)

在构建H2OContextInternalH2OBackend时会出现异常。

我会问如何在IDE

中运行的spark local模式下运行此示例

0 个答案:

没有答案