火花壳 - 主干纱卡住了

Question

我通过Homebrew安装了Hadoop和Spark

$ brew list --versions | grep spark
apache-spark 2.2.0
$ brew list --versions | grep hadoop
hadoop 2.8.1 2.8.2 hdfs

我正在使用Hadoop 2.8.2。

我按照this post配置了Hadoop。另外，按照this post将spark.yarn.archive配置为：

spark.yarn.archive                 hdfs://localhost:9000/user/panc25/spark-jars.zip

以下是我.bash_profile中的Hadoop / Spark相关环境设置：

# ---------------------
# Hadoop
# ---------------------
export HADOOP_HOME=/usr/local/Cellar/hadoop/2.8.2
export YARN_CONF_DIR=$HADOOP_HOME/libexec/etc/hadoop/
alias hadoop-start="$HADOOP_HOME/sbin/start-dfs.sh;$HADOOP_HOME/sbin/start-yarn.sh"
alias hadoop-stop="$HADOOP_HOME/sbin/stop-yarn.sh;$HADOOP_HOME/sbin/stop-dfs.sh"
# ---------------------
# Apache Spark
# ---------------------
export SPARK_HOME=/usr/local/Cellar/apache-spark/2.2.0/libexec
export PATH=$SPARK_HOME/../bin:$SPARK_HOME/sbin:$PATH

我可以成功启动hadoop（hdfa + yarn）：

$ hadoop-start
17/11/12 17:08:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /usr/local/Cellar/hadoop/2.8.2/libexec/logs/hadoop-panc25-namenode-mbp13mid2017.local.out
localhost: starting datanode, logging to /usr/local/Cellar/hadoop/2.8.2/libexec/logs/hadoop-panc25-datanode-mbp13mid2017.local.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/Cellar/hadoop/2.8.2/libexec/logs/hadoop-panc25-secondarynamenode-mbp13mid2017.local.out
17/11/12 17:08:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /usr/local/Cellar/hadoop/2.8.2/libexec/logs/yarn-panc25-resourcemanager-mbp13mid2017.local.out
localhost: starting nodemanager, logging to /usr/local/Cellar/hadoop/2.8.2/libexec/logs/yarn-panc25-nodemanager-mbp13mid2017.local.out
$ jps
92723 NameNode
93188 Jps
93051 ResourceManager
93149 NodeManager
92814 DataNode
92926 SecondaryNameNode

然而，当我开始spark-shell --master yarn时，它似乎冻结了，我不知道发生了什么：

有什么问题？

顺便说一下，我可以访问SparkUI http://localhost:4040/，但所有页面都是空白的。

Answer 1

我遇到了类似的问题，原因是我忘了追加/ conf到HADOOP_CONF_DIR env变量（/ etc / hadoop / conf）。

Answer 2

就我而言，我运行的是 spark 2.1 cloudera 发行版并指定了 HADOOP_CONF_DIR=/etc/hadoop/conf/:/etc/hive/conf/ 。由于某种原因它卡住了，所以我将它修改为 HADOOP_CONF_DIR=/etc/hadoop/conf/ 并且它起作用了。还在寻找根本原因！