Question

我有5个节点的spark集群，其中2个节点正在运行master。在HA（由Zookeeper）场景中，任何人都将被选为主人。

at the time of submitting application using command

/bin/spark-submit  --class SparkAggregator.java --deploy-mode cluster --supervise --master spark://host1:7077

getting error 
Can only accept driver submissions in ALIVE state. Current state: STANDBY.

spark-submit doe not allow multiple master name in --master.

Question:
How to identify the elected master at the time of spark-submit.

Thanks
Pankaj

Answer 1

主选项可以使用多个火花主控器，因此如果您有多个列表，则它们之间使用逗号。 e.g。

/ bin / spark-submit --class SparkAggregator.java --deploy-mode cluster --supervise --master spark：// host1：7077，host2：7077，host3：7077

如果尝试连接所有这些，则会使用第一个响应，这允许您在群集中使用多个主服务器，其中只有一个处于活动状态，其余处于待机状态。

Answer 2

Spark有一个隐藏的API，可以告诉您Spark Cluster的状态

API请求 - http://SPARK_MASTER_IP:8080/json/

输出 -

    {
  "url" : "spark://10.204.216.233:7077",
  "workers" : [ {
    "id" : "worker-20170606104140-10.204.217.96-40047",
    "host" : "10.204.217.96",
    "port" : 40047,
    "webuiaddress" : "http://10.204.217.96:8081",
    "cores" : 4,
    "coresused" : 0,
    "coresfree" : 4,
    "memory" : 29713,
    "memoryused" : 0,
    "memoryfree" : 29713,
    "state" : "ALIVE",
    "lastheartbeat" : 1496760671542
  }, {
    "id" : "worker-20170606104144-10.204.219.15-42749",
    "host" : "10.204.219.15",
    "port" : 42749,
    "webuiaddress" : "http://10.204.219.15:8081",
    "cores" : 4,
    "coresused" : 0,
    "coresfree" : 4,
    "memory" : 29713,
    "memoryused" : 0,
    "memoryfree" : 29713,
    "state" : "ALIVE",
    "lastheartbeat" : 1496760675649
  }, {
    "id" : "worker-20170606104151-10.204.217.249-35869",
    "host" : "10.204.217.249",
    "port" : 35869,
    "webuiaddress" : "http://10.204.217.249:8081",
    "cores" : 4,
    "coresused" : 0,
    "coresfree" : 4,
    "memory" : 29713,
    "memoryused" : 0,
    "memoryfree" : 29713,
    "state" : "ALIVE",
    "lastheartbeat" : 1496760682270
  } ],
  "cores" : 12,
  "coresused" : 0,
  "memory" : 89139,
  "memoryused" : 0,
  "activeapps" : [ ],
  "completedapps" : [ ],
  "activedrivers" : [ ],
  "status" : "ALIVE"
}

在火花提交时识别现场火花大师

2 个答案: