Question

在Compute Engine上使用4个实例，每个运行的spark都使用Cloudera Manager进行设置。我在启动master并在本地浏览器中连接时没有问题，它连接为spark：// instance-1：7077。当我在剩下的实例上启动启动从站时，我没有错误，直到我查看日志：

16/05/02 13:10:18 INFO worker.Worker: Started daemon with process name: 12612@instance-2.c.cluster1-1294.internal
16/05/02 13:10:18 INFO worker.Worker: Registered signal handlers for [TERM, HUP, INT]
16/05/02 13:10:18 INFO spark.SecurityManager: Changing view acls to: root
16/05/02 13:10:18 INFO spark.SecurityManager: Changing modify acls to: root
16/05/02 13:10:18 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with mod$
16/05/02 13:10:19 INFO util.Utils: Successfully started service 'sparkWorker' on port 60270.
16/05/02 13:10:19 INFO worker.Worker: Starting Spark worker 10.142.0.3:60270 with 2 cores, 6.3 GB RAM
16/05/02 13:10:19 INFO worker.Worker: Running Spark version 1.6.0
16/05/02 13:10:19 INFO worker.Worker: Spark home: /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/spark
16/05/02 13:10:19 ERROR worker.Worker: Failed to create work directory /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/spark/work

如果我使用mkdir创建'work'，那么它会抛出并出错并说目录已经存在：

mkdir: cannot create directory ‘work’: File exists

该文件确实存在，当使用ls查找时，它以红色突出显示，背景为黑色。任何帮助将不胜感激。

Answer 1

也许这是许可问题，

试试这个，

   $sudo chown -R your_userName:your_groupName /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/spark

现在更改上述路径的模式

  $sudo chmod 777 /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/spark

此外，所有奴隶必须互相拥有ssh并且能够互相交谈。

并将spark的所有配置文件复制到从节点。

无法在Spark中将从站连接到主站

1 个答案: