由于在运行时找不到某些NoClassDefFoundError
,我试图摆脱一些jars
。所以我在我的hdfs
系统中添加了一些lib,然后我打电话给我,
String lib = "/path/to/lib";
Path hdfsJar = new Path(lib);
DistributedCache.addFileToClassPath(hdfsJar, conf);
现在,我仍然收到错误。但是,如果我在jars
中设置$HADOOP_CLASSPATH
。我对DistributedCache
电话做错了吗?
编辑:
java.lang.RuntimeException: java.lang.NoClassDefFoundError: gov/nih/nlm/nls/metamap/MetaMapApi
at org.apache.hadoop.mapreduce.lib.chain.Chain.joinAllThreads(Chain.java:526)
at org.apache.hadoop.mapreduce.lib.chain.ChainMapper.run(ChainMapper.java:169)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.NoClassDefFoundError: gov/nih/nlm/nls/metamap/MetaMapApi
at org.avrosation.metamap.ChainMetaProcess$TokenizerMapper.map(ChainMetaProcess.java:25)
at org.avrosation.metamap.ChainMetaProcess$TokenizerMapper.map(ChainMetaProcess.java:16)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapreduce.lib.chain.Chain$MapRunner.run(Chain.java:321)
Caused by: java.lang.ClassNotFoundException: gov.nih.nlm.nls.metamap.MetaMapApi
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
答案 0 :(得分:0)
尝试提供完全合格的HDFS路径。尝试以下代码:
确保将jar上传到HDFS(hdfs上的任何位置,我假设是/ tmp)。
hadoop fs -copyFromLocal my.jar /tmp
然后编辑你的java代码,如:
String lib = "hdfs://localhost:9000/tmp/my.jar";
Path hdfsJar = new Path(lib);
DistributedCache.addFileToClassPath(hdfsJar, conf);
此分发缓存的详细信息:https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/filecache/DistributedCache.html
答案 1 :(得分:0)
事实上,我从一开始就怀疑我ide
IntelliJ Idea 14
的一些问题,因为我不得不处理代码的重大重构,然后让代码揭示问题。首先,我尝试清理建筑物没有成功,然后我只是创建了一个其他项目并复制粘贴了类和库的导入,这就成了伎俩!