Question

我已经通过参考https://github.com/MuhammadBilalYar/Hadoop-On-Window/wiki/Step-by-step-Hadoop-2.8.0-installation-on-Window-10

在Windows 7上配置了hadoop。

正在运行map reduce程序（已成功编译）时，出现以下完全错误

18/10/03 20:04:21 INFO util.NativeCodeLoader: Loaded the native-hadoop library
18/10/03 20:04:21 ERROR nativeio.NativeIO: Unable to initialize NativeIO libraries
java.lang.NoClassDefFoundError: org/apache/hadoop/io/nativeio/NativeIO$POSIX$Stat
    at org.apache.hadoop.io.nativeio.NativeIO.initNative(Native Method)
    at org.apache.hadoop.io.nativeio.NativeIO.<clinit>(NativeIO.java:89)
    at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:655)
    at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)
    at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:193)
    at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
    at com.example.hadoop.textToparquet.TextParquetConverter.run(TextParquetConverter.java:41)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at com.example.hadoop.textToparquet.TextParquetConverter.main(TextParquetConverter.java:22)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.nativeio.NativeIO$POSIX$Stat
    at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 18 more
18/10/03 20:04:21 ERROR security.UserGroupInformation: PriviledgedActionException as:vxr83 cause:java.io.IOException: Failed to set permissions of path: \tmp\hadoop-vxr83\mapred\staging\vxr831620787823\.staging to 0700
Exception in thread "main" java.io.IOException: Failed to set permissions of path: \tmp\hadoop-vxr83\mapred\staging\vxr831620787823\.staging to 0700
    at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:691)
    at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:664)
    at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)
    at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:193)
    at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
    at com.example.hadoop.textToparquet.TextParquetConverter.run(TextParquetConverter.java:41)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at com.example.hadoop.textToparquet.TextParquetConverter.main(TextParquetConverter.java:22)

请帮助

Answer 1

问题1：- java.io.IOException：无法将路径\ tmp \ hadoop-vxr83 \ mapred \ staging \ vxr831620787823.staging设置为0700

解决方案：-

从以下URL下载patch-hadoop_7682-1.0.x-win，并将其放在C：\ hadoop-2.7.2 \ lib \下 https://github.com/congainc/patch-hadoop_7682-1.0.x-win
将此jar文件添加到Eclipse Java项目类路径中
在程序中，添加以下配置：

job.getConfiguration（）。set（“ fs.file.impl”，“ com.conga.services.hadoop.patch.HADOOP_7682.WinLocalFileSystem”）;

问题2：-错误nativeio.NativeIO：无法初始化NativeIO库

解决方案：-将vm arugment添加到Eclipse中的map reduce程序的运行配置中，作为-Djava.library.path = C：\ hadoop-2.7.2 \ lib \ native

Answer 2

当我使用不正确的罐子时得到了这个。我曾经在以下位置修复它的那些

    <!-- parquet parquet parquet parquet parquet - START -->
<dependency>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro</artifactId>
  <version>1.9.2</version>
</dependency>
<dependency>
  <groupId>org.apache.parquet</groupId>
  <artifactId>parquet-avro</artifactId>
  <version>1.11.0</version>
</dependency>

<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-common</artifactId>
  <version>${hadoop.version}</version>
</dependency>
<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-client</artifactId>
  <version>${hadoop.version}</version>
</dependency>
<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-aws</artifactId>
  <version>${hadoop.version}</version>
</dependency>
<!-- parquet parquet parquet parquet parquet - END -->



 <properties>
    <hadoop.version>3.2.1</hadoop.version>
  </properties>

我还使用了hadoop.dll和winutils.exe，但不确定是否需要它们。

出现错误nativeio.NativeIO：在Windows中通过Eclipse运行MapReduce时无法初始化NativeIO库

2 个答案: