Question

我正在尝试在新的ubuntu 18.10 VM上设置Spark（具体来说是pySpark）。我已经设置了Hadoop，没有密码的SSH（本地主机），名为hduser的其他用户，这是Hadoop需要的。现在，我在其旁边放置了Spark文件夹，为bashrc添加了路径：

export HADOOP_HOME=/usr/local/hadoop-3.1.2
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export YARN_HOME=${HADOOP_HOME}
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
# Native path
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
# Java path
export JAVA_HOME="/usr"
# OS path
export PATH=$PATH:$HADOOP_HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/sbin

export SPARK_HOME=/usr/local/spark-2.4.1-bin-hadoop2.7
export PATH=$PATH:/usr/local/spark-2.4.1-bin-hadoop2.7/bin

所以现在从hduser中，我进入$ SPARK_HOME并尝试运行./sbin/start-all.sh，然后得到以下消息：

chown: changing ownership of '/usr/local/spark-2.4.1-bin-hadoop2.7/logs': Operation not permitted
starting org.apache.spark.deploy.master.Master, logging to /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.master.Master-1-cristi-VirtualBox.out
/usr/local/spark-2.4.1-bin-hadoop2.7/sbin/spark-daemon.sh: line 128: /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.master.Master-1-cristi-VirtualBox.out: Permission denied
failed to launch: nice -n 0 /usr/local/spark-2.4.1-bin-hadoop2.7/bin/spark-class org.apache.spark.deploy.master.Master --host cristi-VirtualBox --port 7077 --webui-port 8080
tail: cannot open '/usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.master.Master-1-cristi-VirtualBox.out' for reading: No such file or directory
full log in /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.master.Master-1-cristi-VirtualBox.out
localhost: chown: changing ownership of '/usr/local/spark-2.4.1-bin-hadoop2.7/logs': Operation not permitted
localhost: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-cristi-VirtualBox.out
localhost: /usr/local/spark-2.4.1-bin-hadoop2.7/sbin/spark-daemon.sh: line 128: /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-cristi-VirtualBox.out: Permission denied
localhost: failed to launch: nice -n 0 /usr/local/spark-2.4.1-bin-hadoop2.7/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://cristi-VirtualBox:7077
localhost: tail: cannot open '/usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-cristi-VirtualBox.out' for reading: No such file or directory
localhost: full log in /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-cristi-VirtualBox.out

到底是什么问题？我不明白发生了什么。

Answer 1

结果是我需要授予其他用户权限。我用了 sudo chown -R hduser /usr/local/spark-2.4.1-bin-hadoop2.7成功了

无法启动spark start-all.sh权限被拒绝

1 个答案: