Question

我正在运行一个应用程序，但无法连接到Spark。这是相关的代码：

System.setProperty("spark.akka.frameSize", "200")
System.setProperty("spark.akka.timeout", "100")
System.setProperty("spark.default.parallelism", "288")
var sparkmaster = "spark://1.17.8.5:7098"
var sc = new SparkContext(sparkmaster, "tool", "/opt/spark")

相关输出是：

13/08/14 17:15:30 INFO storage.BlockManagerUI: Started BlockManager web UI at ----
13/08/14 17:15:30 INFO spark.SparkContext: Added JAR target/scala-2.9.1/pivot-spark_2.9.1-1.0.jar at http://1.17.8.5:50262/jars/tool_2.9.1-1.0.jar with timestamp 1376500530641
13/08/14 17:15:30 INFO cluster.FairSchedulableBuilder: Create default pool with name:default,schedulingMode:FIFO,minShare:2,weight:1
13/08/14 17:15:30 INFO client.Client$ClientActor: Connecting to master spark://1.17.8.5:7098

正如您所看到的，它实际上并未连接到工作人员。但是，如果我只是在终端中运行spark-shell，我会得到：

13/08/14 17:25:53 INFO BlockManagerUI: Started BlockManager web UI at ---
13/08/14 17:25:53 INFO Client$ClientActor: Connecting to master spark://1.17.8.5:7098
Spark context available as sc.
13/08/14 17:25:53 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20130814172553-0029
13/08/14 17:25:53 INFO Client$ClientActor: Executor added: app-20130814172553-0029/0 on worker-20130808012122----42908 with 4 cores
13/08/14 17:25:53 INFO SparkDeploySchedulerBackend: Granted executor ID app-20130814172553-0029/0 on host --- with 4 cores, 8.0 GB RAM
13/08/14 17:25:53 INFO Client$ClientActor: Executor added: app-20130814172553-0029/1 on worker-20130808012122-----59902 --- with 4 cores
...etc

这里有什么问题？

Answer 1

糟糕！事实证明我需要设置$SPARK_HOME似乎自动设置的环境变量spark-shell。

无法以编程方式连接到Spark集群但是spark-shell可以吗？

1 个答案: