Question

My Spark环境

Spark  -> 2.1.0
Hadoop -> 2.8.1
Eclipse -> Neon 2

我在纱线模式下获得火花环境时卡住了。如何在纱线模式下获得火花背景，请帮我解决。

我的Hadoop，Yarn和Spark安装成功。

$ jps
3200 NameNode
5264 ExecutorLauncher
5328 CoarseGrainedExecutorBackend
3555 SecondaryNameNode
5316 CoarseGrainedExecutorBackend
7590 Jps
3357 DataNode
4045 NodeManager
5118 SparkSubmit
3727 ResourceManager

我在纱线模式下获取火花背景的源代码： -

public class JavaClient {

    public static void main(String[] args) {
        // TODO Auto-generated method stub
        SparkConf conf = new SparkConf().setAppName("SparkTest").setMaster("yarn-client");
        SparkSession spark = SparkSession.builder().config(conf).getOrCreate();


        System.out.println(spark.version() + " : " + spark.sparkContext());

    }
}

输出： -

17/09/22 10:24:11 INFO Client: Application report for application_1506052073594_0011 (state: ACCEPTED)

17/09/22 10:24:12 INFO Client: Application report for application_1506052073594_0011 (state: ACCEPTED) 

17/09/22 10:24:13 INFO Client: Application report for application_1506052073594_0011 (state: ACCEPTED)

.
.
.

等等，没有返回火花会话。

Answer 1

在资源管理器UI（localhost：8088）中检查您是否有可用于Application Master和执行程序的内存/ CPU插槽

Answer 2

尝试“纱线”代替“纱线客户”

public class JavaClient {

public static void main(String[] args) {
    // TODO Auto-generated method stub
    SparkConf conf = new SparkConf().setAppName("SparkTest").setMaster("yarn");
    SparkSession spark = SparkSession.builder().config(conf).getOrCreate();


    System.out.println(spark.version() + " : " + spark.sparkContext());

}

Spark Hang输出状态：YARN MODE中的ACCEPTED

2 个答案: