Datanode守护进程不在datanodes hadoop上启动

时间：2013-12-06 19:29:14

标签： hadoop bigdata

我无法在群集上启动datanode守护程序（版本v2.2）。它在主节点中启动很好但只是不在数据节点中启动。没有在数据节点上创建日志文件，它们是在主节点守护程序中创建的，并且没有错误消息。我已经确定下面的事情是正确的。

  1. 我能够从master使用密码ssh所有数据节点。我还将HADOOP_SECURE_DN_USER用户设置为“hadoop”，这是我计划在所有节点上启动数据节点的用户。
  2. 我已将数据节点添加到从属文件，每行一个。
  3. HADOOP_HOME（/home/hadoop/hadoop-2.2.0），HADOOP_CONF_DIR（$ HADOOP_HOME / etc / hadoop）在所有节点上设置。
  4. 所有必需的目录都存在于datanode上，用户已创建，ipv6已禁用
  5. 添加了必要的配置文件参数，它们如下 -

    6. 以下是日志文件以供参考。他们没有任何错误。注意“网络拓扑具有0个机架和0个数据节点”，表明它无法识别所有数据节点（可能是安全模式1，不确定）。非常感谢任何帮助。

    芯-site.xml中

    <configuration>
<property>
   <name>fs.default.name</name>
   <value>hdfs://localhost:9000</value>
</property>

<property>
   <name>io.file.buffer.size</name>
   <value>131072</value>
</property>
</configuration>

    HDFS-site.xml中

    <configuration>
<property>
   <name>dfs.replication</name>
   <value>2</value>
 </property>
 <property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/home/hadoop/namenode</value>
 </property>
 <property>
   <name>dfs.datanode.data.dir</name>
   <value>file:/home/hadoop/datanode</value>
 </property>
</configuration>

    纱-site.xml中

    <property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
</property>
<property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
 <property>
   <name>yarn.nodemanager.log.dirs</name>
   <value>/home/yarn/logs</value>
 </property>
</configuration>

    mapred-site.xml中

    <configuration>
   <property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
   </property>

  <property>
      <name>mapreduce.task.io.sort.mb</name>
      <value>1024</value>
   </property>

</configuration>

    Namenode日志：

    2013-12-06 23:54:46,940 INFO org.apache.hadoop.hdfs.StateChange: STATE* Leaving safe mode after 1 secs
2013-12-06 23:54:46,940 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 0 racks and 0 datanodes
2013-12-06 23:54:46,940 INFO org.apache.hadoop.hdfs.StateChange: STATE* UnderReplicatedBlocks has 0 blocks
2013-12-06 23:54:46,972 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2013-12-06 23:54:46,972 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9000: starting
2013-12-06 23:54:46,975 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode RPC up at: localhost/192.168.56.1:9000
2013-12-06 23:54:46,975 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services required for active state
2013-12-06 23:55:08,530 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* registerDatanode: from DatanodeRegistration(192.168.56.1, storageID=DS-1268869381-192.168.56.1-50010-1386350725676, infoPort=50075, ipcPort=50020, storageInfo=lv=-47;cid=CID-d6194959-5a13-4d8b-8428-25134e8fb746;nsid=2144581313;c=0) storage DS-1268869381-192.168.56.1-50010-1386350725676
2013-12-06 23:55:08,535 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.56.1:50010
2013-12-06 23:55:08,717 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: BLOCK* processReport: Received first block report from 192.168.56.1:50010 after starting up or becoming active. Its block contents are no longer considered stale
2013-12-06 23:55:08,718 INFO BlockStateChange: BLOCK* processReport: from DatanodeRegistration(192.168.56.1, storageID=DS-1268869381-192.168.56.1-50010-1386350725676, infoPort=50075, ipcPort=50020, storageInfo=lv=-47;cid=CID-d6194959-5a13-4d8b-8428-25134e8fb746;nsid=2144581313;c=0), blocks: 0, processing time: 2 msecs

    Datanode Log（在主节点上）：

    2013-12-06 23:55:08,469 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding block pool BP-1981795271-192.168.56.1-1386350567299
2013-12-06 23:55:08,470 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-1981795271-192.168.56.1-1386350567299 on volume /home/hadoop/datanode/current...
2013-12-06 23:55:08,479 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time taken to scan block pool BP-1981795271-192.168.56.1-1386350567299 on /home/hadoop/datanode/current: 8ms
2013-12-06 23:55:08,479 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Total time to scan all replicas for block pool BP-1981795271-192.168.56.1-1386350567299: 9ms
2013-12-06 23:55:08,479 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding replicas to map for block pool BP-1981795271-192.168.56.1-1386350567299 on volume /home/hadoop/datanode/current...
2013-12-06 23:55:08,479 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time to add replicas to map for block pool BP-1981795271-192.168.56.1-1386350567299 on volume /home/hadoop/datanode/current: 0ms
2013-12-06 23:55:08,479 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Total time to add all replicas to map: 0ms
2013-12-06 23:55:08,485 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1981795271-192.168.56.1-1386350567299 (storage id DS-1268869381-192.168.56.1-50010-1386350725676) service to localhost/192.168.56.1:9000 beginning handshake with NN
2013-12-06 23:55:08,560 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool Block pool BP-1981795271-192.168.56.1-1386350567299 (storage id DS-1268869381-192.168.56.1-50010-1386350725676) service to localhost/192.168.56.1:9000 successfully registered with NN
2013-12-06 23:55:08,560 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: For namenode localhost/192.168.56.1:9000 using DELETEREPORT_INTERVAL of 300000 msec  BLOCKREPORT_INTERVAL of 21600000msec Initial delay: 0msec; heartBeatInterval=3000
2013-12-06 23:55:08,674 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Namenode Block pool BP-1981795271-192.168.56.1-1386350567299 (storage id DS-1268869381-192.168.56.1-50010-1386350725676) service to localhost/192.168.56.1:9000 trying to claim ACTIVE state with txid=5
2013-12-06 23:55:08,674 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Acknowledging ACTIVE Namenode Block pool BP-1981795271-192.168.56.1-1386350567299 (storage id DS-1268869381-192.168.56.1-50010-1386350725676) service to localhost/192.168.56.1:9000
2013-12-06 23:55:08,767 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0 blocks took 2 msec to generate and 90 msecs for RPC and NN processing
2013-12-06 23:55:08,767 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: sent block report, processed command:org.apache.hadoop.hdfs.server.protocol.FinalizeCommand@38568c24
2013-12-06 23:55:08,773 INFO org.apache.hadoop.util.GSet: Computing capacity for map BlockMap
2013-12-06 23:55:08,773 INFO org.apache.hadoop.util.GSet: VM type       = 64-bit
2013-12-06 23:55:08,773 INFO org.apache.hadoop.util.GSet: 0.5% max memory = 889 MB
2013-12-06 23:55:08,773 INFO org.apache.hadoop.util.GSet: capacity      = 2^19 = 524288 entries
2013-12-06 23:55:08,774 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Periodic Block Verification Scanner initialized with interval 504 hours for block pool BP-1981795271-192.168.56.1-1386350567299
2013-12-06 23:55:08,778 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Added bpid=BP-1981795271-192.168.56.1-1386350567299 to blockPoolScannerMap, new size=1

0 个答案:

没有答案
相关问题
最新问题