Question

我在新安装的Ubuntu服务器上设置了一个新的独立Apache Spark服务器。我试着把我的第一份工作送到那里，但我并不是很成功。

以下是我在当地做的事情：

    SparkConf conf = new SparkConf().setAppName("myFirstJob").setMaster("local[*]");
    JavaSparkContext javaSparkContext = new JavaSparkContext(conf);
    javaSparkContext.setLogLevel("WARN");
    SQLContext sqlContext = new SQLContext(javaSparkContext);

    System.out.println("Hello, Remote Spark v." + javaSparkContext.version());

    DataFrame df;
    df = sqlContext.read().option("dateFormat", "yyyy-mm-dd")
            .json("./src/main/resources/north-carolina-school-performance-data.json");
    df = df.withColumn("district", df.col("fields.district"));
    df = df.groupBy("district").count().orderBy(df.col("district"));
    df.show(150);

它有效：它显示北卡罗来纳州学区的名称和学区的学校数量：

Hello, Remote Spark v.1.6.1
+--------------------+-----+
|            district|count|
+--------------------+-----+
|Alamance-Burlingt...|   34|
|Alexander County ...|   10|
|Alleghany County ...|    4|
|Anson County Schools|   10|
| Ashe County Schools|    5|
|Asheboro City Sch...|    8|
...

现在，如果我将第一行更改为：

SparkConf conf = new SparkConf().setAppName("myFirstJob").setMaster("spark://10.0.100.120:7077");

很顺利：

Hello, Remote Spark v.1.6.1
16/07/12 10:58:34 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
16/07/12 10:58:49 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

第一件奇怪的事（对我来说）是服务器有Spark 1.6.2。我有点期待看到1.6.2作为版本号。

然后在用户界面上，我去那里看看：

如果我点击app-20160712105816-0011，我会得到：

点击其他链接会将我带到我当地的Apache Spark实例。

点击后，我可以看到类似的内容：

如果我查看服务器上的日志，我会看到：

16/07/12 10:37:00 INFO Master: Registered app myFirstJob with ID app-20160712103700-0009
16/07/12 10:37:03 INFO Master: Received unregister request from application app-20160712103700-0009
16/07/12 10:37:03 INFO Master: Removing app app-20160712103700-0009
16/07/12 10:37:03 INFO Master: 10.0.100.100:54396 got disassociated, removing it.
16/07/12 10:37:03 INFO Master: 10.0.100.100:54392 got disassociated, removing it.
16/07/12 10:50:44 INFO Master: Registering app myFirstJob
16/07/12 10:50:44 INFO Master: Registered app myFirstJob with ID app-20160712105044-0010
16/07/12 10:51:20 INFO Master: Received unregister request from application app-20160712105044-0010
16/07/12 10:51:20 INFO Master: Removing app app-20160712105044-0010
16/07/12 10:51:20 INFO Master: 10.0.100.100:54682 got disassociated, removing it.
16/07/12 10:51:20 INFO Master: 10.0.100.100:54680 got disassociated, removing it.
16/07/12 10:58:16 INFO Master: Registering app myFirstJob
16/07/12 10:58:16 INFO Master: Registered app myFirstJob with ID app-20160712105816-0011

对我来说这一切似乎都很好......

我在Apache Spark Server installation requires Hadoop? Not automatically installed?有一个前一个问题（未解决），环境相同，但这是一个完全不同的 - 更小 - 应用程序。

有关正在发生的事情的任何线索？

Answer 1

根据Web UI屏幕截图，您的服务器没有worker（slave）。 Spark提供了一些脚本来启动集群。

sbin / start-all.sh：启动主人和conf/slaves
sbin / start-slaves.sh：仅启动conf/slaves
sbin / start-master.sh：在当前计算机中启动主服务器
sbin / start-slave.sh：在当前计算机中启动slave

如果您的群集配置正确，则只需在主计算机上调用start-all.sh即可启动所有内容。

Apache Spark（Java）上的远程作业

1 个答案: