我有一个scala 2.10 spark 1.5.0 sbt app我正在eclipse中开发。在我的主要方法中,我有:
val sc = new SparkContext("local[2]", "My Spark App")
// followed by operations to do things with the spark context to count up lines in a file
当我在eclipse IDE中运行此应用程序时,它可以工作并输出我期望的结果,但是当我使用以下命令更改spark上下文以连接到我的集群时:
val master = "spark://My-Hostname-From-The-Spark-Master-Page.local:7077"
val conf = new SparkConf().setAppName("My Spark App").setMaster(master)
val sc = new SparkContext(conf)
我收到的错误如下:
WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkMaster@My-Hostname-From-The-Spark-Master-Page.local:7077] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
是什么给出的?如何使用我启动的现有主节点和从节点运行我的工作?我知道spark-submit是推荐的,但是像Zepplin和Notebook这样的应用程序设计用于使用spark而不需要" spark-submit"?