Question

我正在尝试从远程HDFS系统读取文件并显示在本地计算机的控制台中。请注意，本地计算机只能通过.pem文件形式的SSH密钥与任何HDFS节点建立连接。

当我执行下面显示的代码时，程序运行，保持空闲一段时间，最后显示：

BlockMissingException : Could not obtain block

我的代码：

try {
            UserGroupInformation ugi = UserGroupInformation.createRemoteUser("remoteUser");

            ugi.doAs(new PrivilegedExceptionAction<Void>() {
                public Void run() throws Exception {
                    conf = new Configuration();
                    conf.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
                    conf.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName());
                    conf.addResource(new Path("/etc/hadoop/conf/core-site.xml"));
                    conf.addResource(new Path("/etc/hadoop/conf/hdfs-site.xml"));
                    conf.addResource(new Path("/etc/hadoop/conf/mapred-site.xml"));
                    conf.set("fs.default.name", hdfsurl);
                    conf.set("fs.defaultFS", hdfsurl);
                    conf.set("hadoop.job.ugi", "remoteUser");
                    conf.set("hadoop.ssl.enabled", "false");
                    readFromHDFS(hdfsurl);
                    return null;
                }
            });
        } catch (Exception e) {



public static void readFromHDFS(String hdfsURL) throws Exception {
        FileSystem fileSystem = FileSystem.get(conf);
        Path path = new Path(hdfsURL);
        if (!fileSystem.exists(path)) {
            System.out.println("File does not exists");
            return;
        }
        FSDataInputStream in = fileSystem.open(path);
        Scanner sc = new Scanner(in);
        while (sc.hasNextLine()) {
            System.out.println("line read from hdfs...." + sc.nextLine());
        }
        in.close();
        fileSystem.close();
}

Answer 1

1）输入hadoop fsck HDFS_FILE 检查特定的hdfs文件是否健康如果不是，则特定文件已损坏。删除损坏的文件并尝试下面的命令

2）输入hadoop dfsadmin -report 检查Missing blocks: 0

的值

从远程HDFS读取文件

1 个答案: