Question

我正在eclipse中构建我的MapReduce程序，并试图将jar作为一个可运行的jar导出我自己的运行配置。在运行配置中，我指定主类并获取导出的jar并将其放入我的hadoop / playground文件夹中。当我尝试使用此命令执行它时：

hadoop jar playground/MyProg.jar "packageinfo" "input" "output"

我总是得到一个PrivilegedAccessException，说输入路径不存在。我检查了包结构和“输入”＆amp; “输出”路径。我怀疑它与导出的jar的结构方式有关。

使用以下两个命令非常不方便：

$ javac -classpath *:lib/* -d playground/classes playground/src/WordCount.java
$ jar -cvf playground/WordCount.jar -C playground/classes/ .

尤其如此，因为我的MapReduce程序的复杂性增加了。

我想知道是否有一种特定的方法可以从Eclipse导出具有正确目录结构的jar，这样我就不会得到PrivilegedAccessException。对我来说，MapReduce本身也很容易开发 - 调试，部署和扩展测试它是一个巨大的挑战，特别是在Linux环境中，所有的终端和文件系统命令是如此神秘。我非常感谢所有帮助我理解这一过程的输入。

当我尝试运行名为MyJob.jar的应用程序时出现示例错误消息如下：

 hadoop jar playground/MyJob.jar src.main.myjob.MyJob /usr/root/MyJob/cite75_99.txt /usr/root/MyJob/cite75_99OUT.txt
Warning: $HADOOP_HOME is deprecated.

13/09/28 15:20:13 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/09/28 15:20:14 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/09/28 15:20:14 WARN snappy.LoadSnappy: Snappy native library not loaded
13/09/28 15:20:14 INFO mapred.JobClient: Cleaning up the staging area hdfs://localhost:9000/tmp/hadoop-root/mapred/staging/root/.staging/job_201309281443_0007
13/09/28 15:20:14 ERROR security.UserGroupInformation: PriviledgedActionException as:root cause:org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://localhost:9000/user/root/src.main.myjob.MyJob
Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://localhost:9000/user/root/src.main.myjob.MyJob
    at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:197)
    at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:208)
    at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:1081)
    at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1073)
    at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:983)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:910)
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1353)
    at src.main.myjob.MyJob.run(MyJob.java:87)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at src.main.myjob.MyJob.main(MyJob.java:94)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:160)

从Eclipse导出Hadoop Jar

0 个答案: