我是hadoop世界的新手。我为我的应用程序开发了自定义输入格式和阅读器。我在运行单元测试时遇到问题。我按照这篇文章开发了我的单元测试:How to do unit testing of custom RecordReader and InputFormat classes?。除了InputFormat类之外,我的单元测试代码与post中的代码完全相同:
Configuration conf = new Configuration(false); conf.set("fs.default.name", "file:///"); File testFile = new File("/tmp/testfile"); Path path = new Path(testFile.getAbsoluteFile().toURI()); FileSplit split = new FileSplit(path, 0, testFile.length(), null); MyFileInputFormat inputFormat = ReflectionUtils.newInstance(MyFileInputFormat.class, conf); TaskAttemptContext context = new TaskAttemptContextImpl(conf, new TaskAttemptID()); try { RecordReader reader = inputFormat.createRecordReader(split, context); reader.initialize(split, context); } catch (Exception e) { }
当我尝试使用" mvn test"运行此测试时,我收到以下错误:
java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration$DeprecationDelta at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at org.apache.hadoop.mapreduce.util.ConfigUtil.addDeprecatedKeys(ConfigUtil.java:54) at org.apache.hadoop.mapreduce.util.ConfigUtil.loadResources(ConfigUtil.java:42) at org.apache.hadoop.mapred.JobConf.(JobConf.java:119) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:274) at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:80) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
任何人都可以帮我理解如何解决这个问题吗?