Question

我正在尝试设置Hadoop多节点群集。

当我启动我的群集时，这是我在控制台中的响应..

hduser@hadoop-master:/usr/local/hadoop$ /usr/local/hadoop/sbin/start-dfs.sh
Starting namenodes on [hadoop-master]
hadoop-master: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hduser-namenode-hadoop-master.out
hadoop-master: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hduser-datanode-hadoop-master.out
hadoop-child: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hduser-datanode-hadoop-child.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hduser-secondarynamenode-hadoop-master.out
hduser@hadoop-master:/usr/local/hadoop$ jps
21079 NameNode
21258 DataNode
21479 SecondaryNameNode
21600 Jps
hduser@hadoop-master:/usr/local/hadoop$ /usr/local/hadoop/sbin/start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hduser-resourcemanager-hadoop-master.out
hadoop-child: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hduser-nodemanager-hadoop-child.out
hadoop-master: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hduser-nodemanager-hadoop-master.out
hduser@hadoop-master:/usr/local/hadoop$ jps
21079 NameNode
21258 DataNode
22117 Jps
21815 NodeManager
21479 SecondaryNameNode
21658 ResourceManager

您可以在hadoop-child计算机中看到正在启动的datanode。

现在，当我尝试获取所有节点信息时。我没有看到所有节点都显示出来。

hduser@hadoop-master:/usr/local/hadoop$ bin/hdfs dfsadmin -report
Configured Capacity: 21103243264 (19.65 GB)
Present Capacity: 17825124352 (16.60 GB)
DFS Remaining: 17821085696 (16.60 GB)
DFS Used: 4038656 (3.85 MB)
DFS Used%: 0.02%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (1):

Name: 127.0.0.1:50010 (localhost)
Hostname: localhost
Decommission Status : Normal
Configured Capacity: 21103243264 (19.65 GB)
DFS Used: 4038656 (3.85 MB)
Non DFS Used: 3278118912 (3.05 GB)
DFS Remaining: 17821085696 (16.60 GB)
DFS Used%: 0.02%
DFS Remaining%: 84.45%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sun Feb 26 17:13:04 UTC 2017

我需要在这里查看主人和孩子的信息。我有1个主节点和1个子节点。

********************修复*******

基于弗兰克，这是如何修复的。

编辑/ etc / config文件并同时提供主要和子ipaddress。

更改了以下2行

127.0.0.1 localhost hadoop-master
961.118.98.183 hadoop-child

到（在两个节点中）

127.0.0.1 localhost 
961.118.99.251 hadoop-master
961.118.98.183 hadoop-child

2.按以下顺序重新启动集群。应重新格式化datanode。

format will remove only the meta, the datanode's data directories will still be using the old namenode's identity which wil cause the datanode to fail(so delete the directories).

Can you please this order.. 
1) stop the cluster 
2) rm -rf /path/to/datanode/data/dir (in both nodes) 
3) hadoop namenode -format
4) START cluste

Answer 1

设置多节点环境时，必须明确定义namenode和resourcemanager地址。

将此属性添加到两个节点中的core-site.xml

<property>
  <name>fs.defaultFS</name>
  <value>hdfs://hadoop-master:8020</value>
</property>

这也是yarn-site.xml，

<property>
  <name>yarn.resourcemanager.hostname</name>
  <value>hadoop-master</value>
</property>

确保在所有节点的/etc/hosts文件中完成IP地址和主机名映射。

注意：如果服务正在运行，请停止它们并在重新启动它们之前添加这些属性。

Hadoop集群中的节点数

1 个答案: