服务器重新启动后，datanode无法启动：InconsistentFSStateException：目录/ dfs / dn处于不一致状态

Question

我有一个Hadoop集群。

我已经重启了其中一台主机。服务器启动后，datanode服务无法在该服务器上启动。

下面的

是角色日志。有什么建议吗？

目录/ dfs / dn具有数据文件，为什么datanode尝试格式化它？

1:08:24.025 AM  INFO    Storage 
Using 2 threads to upgrade data directories (dfs.datanode.parallel.volumes.load.threads.num=2, dataDirs=2)
1:08:24.031 AM  INFO    Storage 
Lock on /dfs/dn/in_use.lock acquired by nodename 86062@nd1
1:08:24.033 AM  WARN    Storage 
Failed to add storage directory [DISK]file:/dfs/dn/
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /dfs/dn is in an inconsistent state: Can't format the storage directory because the current/ directory is not empty.
    at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.checkEmptyCurrent(Storage.java:480)
    at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:585)
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadStorageDirectory(DataStorage.java:279)
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadDataStorage(DataStorage.java:418)
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:397)
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:575)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1561)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1521)
    at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:313)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:219)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
    at java.lang.Thread.run(Thread.java:748)
1:08:24.044 AM  INFO    Storage 
Lock on /mydisk/dfs/dn/in_use.lock acquired by nodename 86062@nd1
1:08:24.082 AM  INFO    Storage 
Analyzing storage directories for bpid BP-323629408-192.168.1.69-1528205532621
1:08:24.082 AM  INFO    Storage 
Locking is disabled for /mydisk/dfs/dn/current/BP-323629408-192.168.1.69-1528205532621
1:08:24.084 AM  INFO    DataNode    
Setting up storage: nsid=1321570183;bpid=BP-323629408-192.168.1.69-1528205532621;lv=-56;nsInfo=lv=-60;cid=cluster22;nsid=1321570183;c=0;bpid=BP-323629408-192.168.1.69-1528205532621;dnuuid=b93607ce-5d92-442c-ad89-d103dadbd880
1:08:24.096 AM  FATAL   DataNode    
Initialization failed for Block pool <registering> (Datanode Uuid b93607ce-5d92-442c-ad89-d103dadbd880) service to namend/192.168.1.69:8022. Exiting. 
org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 1, volumes configured: 2, volumes failed: 1, volume failures tolerated: 0
    at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.<init>(FsDatasetImpl.java:288)
    at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.newInstance(FsDatasetFactory.java:34)
    at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.newInstance(FsDatasetFactory.java:30)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1574)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1521)
    at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:313)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:219)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
    at java.lang.Thread.run(Thread.java:748)
1:08:24.097 AM  WARN    DataNode    
Ending block pool service for: Block pool <registering> (Datanode Uuid b93607ce-5d92-442c-ad89-d103dadbd880) service to namend/192.168.1.69:8022
1:08:24.198 AM  INFO    DataNode    
Removed Block pool <registering> (Datanode Uuid b93607ce-5d92-442c-ad89-d103dadbd880)
1:08:26.199 AM  WARN    DataNode    
Exiting Datanode
1:08:26.200 AM  INFO    ExitUtil    
Exiting with status 0
1:08:26.202 AM  INFO    DataNode    
SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at nd1/192.168.1.117
************************************************************/

这是hdfs fsck的输出：