Question

我打算从HDFS通过Java获取文件

public static void main(String[] args) throws IOException, URISyntaxException 
{ //1. Get the instance of Configuration
   Configuration configuration = new Configuration();
   File workaround = new File(".");
   System.getProperties().put("hadoop.home.dir", workaround.getAbsolutePath());
        new File("./bin").mkdirs();
        new File("./bin/winutils.exe").createNewFile();
        configuration.set("fs.hdfs.impl", 
            org.apache.hadoop.hdfs.DistributedFileSystem.class.getName()
    );
    configuration.set("fs.file.impl",
      org.apache.hadoop.fs.LocalFileSystem.class.getName()
    );

    //2. URI of the file to be read
    URI uri = new URI("hdfs://10.1.7.24:8020/user/hue/sparkinput/tripdata.csv");

    //3. Get the instance of the HDFS
    FileSystem hdfs = FileSystem.get(uri, configuration);

    //4. A reference to hold the InputStream
    InputStream inputStream = null;
    String pathOfFile=uri.toString();
    try{
          //5. Prepare the Path, i.e similar to File class in Java, Path represents file in HDFS
          Path path = new Path(pathOfFile);
          //6. Open a Input Stream to read the data from HDFS
          inputStream = hdfs.open(path);
          //7. Use the IOUtils to flush the data from the file to console

          IOUtils.copyBytes(inputStream, System.out, 4096, false);

    }finally{

         //8. Close the InputStream once the data is read
         IOUtils.closeStream(inputStream);
    }

    /*  Testing testing = new Testing();
    testing.getMongoTables("sales");*/

}

这是例外

log4j:ERROR Could not find value for key log4j.appender.FILE
log4j:ERROR Could not instantiate appender named "FILE".
2018-01-11 16:35:11 WARN  NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/crypto/key/KeyProviderenter code here
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:148)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2397)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
    at Testing.main(Testing.java:172)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.crypto.key.KeyProvider
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 7 more

Answer 1

您的java程序无法找到hadoop libarary，请更正将解决问题的hadoop jar的路径。

此警告显示您的问题

NativeCodeLoader - Unable to load native-hadoop library for your platform

Hadoop NoClassDef hadoop / crypto / key / KeyProvider

1 个答案: