Question

我已遵循此guide使用EC2实例在AWS上创建多节点Spark群集。在我尝试使用以下命令从主EC2实例启动集群之前，所有内容都完美无误（

）

$SPARK_HOME/sbin/start-all.sh

我获得了＃34;权限被拒绝（公钥）＆＃34;如下：

$SPARK_HOME/sbin/start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /usr/local/spark/logs/spark-ubuntu-org.apache.spark.deploy.master.Master-1-ip-172-31-2-17.out
    ec2-34-205-81-113.compute-1.amazonaws.com: Warning: Permanently added 'ec2-34-205-81-113.compute-1.amazonaws.com,172.31.6.154' (ECDSA) to the list of known hosts.
    ec2-34-205-255-52.compute-1.amazonaws.com: Warning: Permanently added 'ec2-34-205-255-52.compute-1.amazonaws.com,172.31.11.106' (ECDSA) to the list of known hosts.
    ec2-34-201-21-89.compute-1.amazonaws.com: Warning: Permanently added 'ec2-34-201-21-89.compute-1.amazonaws.com,172.31.4.124' (ECDSA) to the list of known hosts.
    ec2-34-205-81-113.compute-1.amazonaws.com: Permission denied (publickey).
    ec2-34-205-255-52.compute-1.amazonaws.com: Permission denied (publickey).
    ec2-34-201-21-89.compute-1.amazonaws.com: Permission denied (publickey).

日志文件如下：

Spark Command: /usr/bin/java -cp /usr/local/spark/conf/:/usr/local/spark/jars/* -Xmx1g org.apache.spark.deploy.master.Master 
--host ip-172-31-2-17.ec2.internal --port 7077 --webui-port 8080
========================================
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
18/01/15 22:31:57 INFO Master: Started daemon with process name: 5772@ip-172-31-2-17
18/01/15 22:31:57 INFO SignalUtils: Registered signal handler for TERM
18/01/15 22:31:57 INFO SignalUtils: Registered signal handler for HUP
18/01/15 22:31:57 INFO SignalUtils: Registered signal handler for INT
18/01/15 22:31:58 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... 
using builtin-java classes where applicable
18/01/15 22:31:58 INFO SecurityManager: Changing view acls to: ubuntu
18/01/15 22:31:58 INFO SecurityManager: Changing modify acls to: ubuntu
18/01/15 22:31:58 INFO SecurityManager: Changing view acls groups to:
18/01/15 22:31:58 INFO SecurityManager: Changing modify acls groups to:
18/01/15 22:31:58 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  
with view permissions: Set(ubuntu); groups with view permissions: Set(); users  with modify permissions: 
Set(ubuntu); groups with modify permissions: Set()
18/01/15 22:31:59 INFO Utils: Successfully started service 'sparkMaster' on port 7077.
18/01/15 22:31:59 INFO Master: Starting Spark master at spark://ip-172-31-2-17.ec2.internal:7077
18/01/15 22:31:59 INFO Master: Running Spark version 2.0.2
18/01/15 22:31:59 INFO Utils: Successfully started service 'MasterUI' on port 8080.
18/01/15 22:31:59 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://ec2-34-205-4-1.compute-1.amazonaws.com:8080
18/01/15 22:31:59 INFO Utils: Successfully started service on port 6066.
18/01/15 22:31:59 INFO StandaloneRestServer: Started REST server for submitting applications on port 6066
18/01/15 22:31:59 INFO Master: I have been elected leader! New state: ALIVE

我在网上搜索了这个错误是什么，建议的解决方案是在我的主人身上运行以下内容：

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

也没有运气，它给了我这个：

cat: /home/ubuntu/.ssh/id_rsa.pub: No such file or directory

此外，在指南的最后有人建议：

如果您在启动时遇到“权限被拒绝”问题群集，你可能没有私人的ssh密钥主节点。将私钥移动到主节点后，即可还必须通过指定将私钥指定为默认值 .ssh / config中的关键位置。例如：
IdentityFile ~/.ssh/spark_cluster.pem

但是本教程中没有关于放置或移动ssh密钥的部分，我现在不知道从哪里开始。我不认为我错过了教程中的任何指令。我的所有机器都是Ubuntu 16.04。请帮忙

Answer 1

看起来SSH存在问题。通过将主节点上的公钥复制到新的从节点，解决了之前类似的question。在您的情况下，尝试生成一个密钥（因为它似乎不存在），然后将其复制到所有从属节点。

cd ~/.ssh
ssh-keygen -t rsa
scp ~/.ssh/id_rsa.pub remote_username@host:~/.ssh/authorized_keys

＆＃34;权限被拒绝（公钥）＆＃34;从master启动AWS EC2 Spark多节点集群时

1 个答案: