Oozie工作流程触发Hadoop Map Reduce作业的Java类。我在Eclipse项目中添加了opencsv-2.3.jar和commons-lang-3-3.1 jar依赖项。该项目成功构建,但是当它在Hadoop集群上移动时,即使我的项目包含jar,我也会收到ClassNotFoundError。
由于这是一个正在运行的现有遗留系统,因此我不希望更改环境依赖性。因此,我尝试通过将库添加到classpath来尝试不同的组合,但没有成功。
尝试:java.lang.NoClassDefFoundError: au/com/bytecode/opencsv/CSVReader - Upload File Vaadin
已通过MR客户端maven依赖项进行检查-org.apache.hadoop:hadoop-mapreduce-client-common:2.6.0-cdh5.4.2。
生产环境中的旧版jar运行良好,但是我项目的已编译jar引发如下错误:
oozie syslog:
INFO [uber-SubtaskRunner] org.apache.hadoop.mapreduce.Job: Running job: job_123213123123_35305
INFO [communication thread] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1548794054671_35304_m_000000_0 is : 1.0
INFO [uber-SubtaskRunner] org.apache.hadoop.mapreduce.Job: Job job_123213123123_35305 running in uber mode : false
INFO [uber-SubtaskRunner] org.apache.hadoop.mapreduce.Job: map 0% reduce 0%
INFO [uber-SubtaskRunner] org.apache.hadoop.mapreduce.Job: Task Id : attempt_123213123123_35305_m_000001_0, Status : FAILED
oozie stderr:
Error: java.lang.ClassNotFoundException: au.com.bytecode.opencsv.CSVParser
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
答案 0 :(得分:0)
opencsv-2.3.jar库是作为外部jar从Eclipse Build Path添加的。我必须使用mvn clean并进行构建。最后,使用目标文件夹中的“ * jar-with-dependencies.jar”修复了该问题。