启动Apache Spark集群

时间:2016-10-19 00:56:40

标签: hadoop apache-spark configuration installation

我在我的四节点集群上安装了Hadoop。我还在每个上安装了Apache Spark。我可以在没有密码的情况下从主服务器ssh到每个从服务器。我也可以启动我的主节点。但是,当我尝试使用/opt/spark/sbin/start-all.sh运行Spark时,出现以下错误:

starting org.apache.spark.deploy.master.Master, logging to /opt/spark/logs/spark-hduser-org.apache.spark.deploy.master.Master-1-lebron.out
doublet: chown: changing ownership of ‘/opt/spark/logs’: Operation not permitted
doublet: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-doublet.out
doublet: /opt/spark/sbin/spark-daemon.sh: line 149: /opt/spark/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-doublet.out: Permission denied
kyrie: chown: changing ownership of ‘/opt/spark/logs’: Operation not permitted
kyrie: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-kyrie.out
kyrie: /opt/spark/sbin/spark-daemon.sh: line 149: /opt/spark/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-kyrie.out: Permission denied
lebron: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-lebron.out
jr: chown: changing ownership of ‘/opt/spark/logs’: Operation not permitted
jr: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-jr.out
jr: /opt/spark/sbin/spark-daemon.sh: line 149: /opt/spark/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-jr.out: Permission denied
doublet: failed to launch org.apache.spark.deploy.worker.Worker:
doublet: tail: cannot open ‘/opt/spark/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-doublet.out’ for reading: No such file or directory
doublet: full log in /opt/spark/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-doublet.out
kyrie: failed to launch org.apache.spark.deploy.worker.Worker:
kyrie: tail: cannot open ‘/opt/spark/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-kyrie.out’ for reading: No such file or directory
jr: failed to launch org.apache.spark.deploy.worker.Worker:
kyrie: full log in /opt/spark/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-kyrie.out
jr: tail: cannot open ‘/opt/spark/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-jr.out’ for reading: No such file or directory
jr: full log in /opt/spark/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-jr.out

(我的电脑名为lebron(master),kyrie,jr,doublet(工人))

请帮助!!!

3 个答案:

答案 0 :(得分:3)

启动服务器:

要启动独立主服务器,请执行:

$ ./sbin/start-master.sh

启动一个或多个工人并通过以下方式将他们连接到主人:

$ ./sbin/start-slave.sh <master-spark-URL>

看一下这篇文章:Apache Spark Cluster Installation and Configuration Guide

答案 1 :(得分:1)

看起来像/ opt / spark文件夹没有写入日志文件的权限 给予完全许可: -     sudo chmod -R 777 / opt / spark 并检查/ opt / spark文件夹user:group是否为所有机器(主机和从机都有)具有相同的值,否则运行此命令     sudo chown -R userName:groupName / opt / spark

答案 2 :(得分:0)

在所有节点上配置密码少ssh。