Question

我的本地操作系统是Ubuntu 14.04。我想通过以下示例代码远程连接到我的Hadoop集群并从HDFS读取。我需要阅读的文件有压缩编解码器。

public static void readFromSequenceFile(String filePath){
    Configuration conf = new Configuration();
    conf.set("fs.default.name" , "hdfs://master.cluster:8020");
    Path path = new Path(filePath);
    try(FileSystem fs = FileSystem.get(conf);
        SequenceFile.Reader reader = new SequenceFile.Reader(fs, path, conf)){
        Writable key = (Writable)ReflectionUtils.newInstance(reader.getKeyClass() , conf);
        Writable value = (Writable)ReflectionUtils.newInstance(reader.getValueClass(), conf);
        long position = reader.getPosition();
        while(reader.next(key , value)){
            boolean sync = reader.syncSeen();
            System.out.println(position + " - " + sync);
            position = reader.getPosition();
        }
    }catch (IOException ioe){
        ioe.printStackTrace();
    }
}

但我收到以下错误：

WARN snappy.LoadSnappy：Snappy本机库可用

WARN util.NativeCodeLoader：无法为您的平台加载native-hadoop库...使用适用的builtin-java类

WARN snappy.LoadSnappy：未加载Snappy本机库

线程中的异常＆＃34; main＆＃34; java.lang.RuntimeException：本机snappy库不可用

从远程hdfs读取一个序列文件（带有snappy压缩）？

0 个答案: