Question

我使用以下命令在Amazon EMR上创建群集：

*./elastic-mapreduce --create --alive --num-instances 1 --instance-type m1.medium --name "CLUSTER_NAME" --hive-interactive --hive-versions 0.13 --hive-site s3://mybucket/hive-site.xml --ami-version 3.3 --subnet subnet-mysubnetid*

几分钟后，我可以运行Hive，一切都很顺利。但是，当我尝试使用以下代码启动具有两个节点的集群时：

*./elastic-mapreduce --create --alive --num-instances 2 --master-instance-type m1.medium  --slave-instance-type m1.medium --name "CLUSTER_NAME" --hive-interactive --hive-versions 0.13 --hive-site s3://mybucket/hive-site.xml --ami-version 3.3 --subnet subnet-mysubnetid*

群集未完成设置。大约50分钟后，群集终止。

我找到了错误日志

xxx.xxx.0.28:40908 Call#2 Retry#0: error: java.io.IOException: File /mnt/var/em/verify-hdfs-access-1147553264033 could only be replicated to 0 nodes instead of minReplication (=1).  There are 0 datanode(s) running and no node(s) are excluded in this operation.

java.io.IOException: File /mnt/var/em/verify-hdfs-access-1147553264033 could only be replicated to 0 nodes instead of minReplication (=1).  There are 0 datanode(s) running and no node(s) are excluded in this operation.
    at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2694)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:599)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

我知道群集无法创建任何DataNode，也许配置有问题。然而，当我配置单节点集群时，一切都很顺利。我已经阅读了很多东西，但此时我有点失落。希望任何人都可以帮助或给我一个解决问题的线索。

Tnanks提前，

巴勃罗

尝试启动具有2个节点的群集时，EMR群集将终止

0 个答案: