Question

我在同一台jvm中运行“ namenode”和“ datanode”，当我尝试写入数据时，出现以下异常

org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy $ NotEnoughReplicasException：在org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom（BlockPlacementPolicyDefault.java:836）在org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom（BlockPlacementPolicyDefault.java:724）在org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalRack（BlockPlacementPolicyDefault.java:631）在org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalStorage（BlockPlacementPolicyDefault.java:591）在org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargetInOrder（BlockPlacementPolicyDefault.java:490）在org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget（BlockPlacementPolicyDefault.java:421）在org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget（BlockPlacementPolicyDefault.java:297）在org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget（BlockPlacementPolicyDefault.java:148）在org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget（BlockPlacementPolicyDefault.java:164）在org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock（BlockManager.java:2127）在org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock（FSDirWriteFileOp.java:294）在org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock（FSNamesystem.java:2771）在org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock（NameNodeRpcServer.java:876）在org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock（ClientNamenodeProtocolServerSideTranslatorPB.java:567）在org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos $ ClientNamenodeProtocol $ 2.callBlockingMethod（ClientNamenodeProtocolProtos.java）在org.apache.hadoop.ipc.ProtobufRpcEngine $ Server $ ProtoBufRpcInvoker.call（ProtobufRpcEngine.java:524）在org.apache.hadoop.ipc.RPC $ Server.call（RPC.java:1025）在org.apache.hadoop.ipc.Server $ RpcCall.run（Server.java:876）在org.apache.hadoop.ipc.Server $ RpcCall.run（Server.java:822）在java.base / java.security.AccessController.doPrivileged（本机方法）在java.base / javax.security.auth.Subject.doAs（Subject.java:423）在org.apache.hadoop.security.UserGroupInformation.doAs（UserGroupInformation.java:1730）在org.apache.hadoop.ipc.Server $ Handler.run（Server.java:2682）

        final File file = new File("C:\\ManageEngine\\test\\data\\namenode");
        final File file1 = new File("C:\\ManageEngine\\test\\data\\datanode1");
        BasicConfigurator.configure();
        final HdfsConfiguration nameNodeConfiguration = new HdfsConfiguration();
        FileSystem.setDefaultUri(nameNodeConfiguration, "hdfs://localhost:5555");
        nameNodeConfiguration.set(DFSConfigKeys.DFS_NAMENODE_NAME_DIR_KEY, file.toURI().toString());
        nameNodeConfiguration.set(DFSConfigKeys.DFS_REPLICATION_KEY, "1" );
        final NameNode nameNode = new NameNode(nameNodeConfiguration);



        final HdfsConfiguration dataNodeConfiguration1 = new HdfsConfiguration();
        dataNodeConfiguration1.set(DFSConfigKeys.DFS_DATANODE_DATA_DIR_KEY, file1.toURI().toString());
        dataNodeConfiguration1.set(DFSConfigKeys.DFS_DATANODE_ADDRESS_KEY, "localhost:5556" );
        dataNodeConfiguration1.set(DFSConfigKeys.DFS_REPLICATION_KEY, "1" );
        FileSystem.setDefaultUri(dataNodeConfiguration1, "hdfs://localhost:5555");
        final DataNode dataNode1 = DataNode.instantiateDataNode(new String[]{}, dataNodeConfiguration1);



        final FileSystem fs = FileSystem.get(dataNodeConfiguration1);

        Path hdfswritepath = new Path(fileName);
        if(!fs.exists(hdfswritepath)) {
            fs.create(hdfswritepath);
            System.out.println("Path "+hdfswritepath+" created.");
        }
        System.out.println("Begin Write file into hdfs");

        FSDataOutputStream outputStream=fs.create(hdfswritepath);
        //Cassical output stream usage
        outputStream.writeBytes(fileContent);
        outputStream.close();
        System.out.println("End Write file into hdfs");

Request data - Image

Answer 1

您的副本数不能超过数据节点的数目。

如果要在单个节点上运行，请在hdfs-site.xml中将dfs.replication设置为1。

无法在HDFS“数据节点”中写入数据-已在排除列表中添加了节点

1 个答案: