Question

我一直在我的项目中使用spring数据hadoop，并且能够在hadoop 1.x中运行distcp作业。最近我们升级到hadoop 2.x，为此我将spring-data-hadoop升级到2.0.4。大多数东西仍在工作，但我遇到了一些distcp的问题。看起来像spring数据hadoop就像这样调用distcp

Class<org.apache.hadoop.tools.DistCp> cl = org.apache.hadoop.tools.DistCp.class;
Class<?> argClass = ClassUtils.resolveClassName("org.apache.hadoop.tools.DistCp$Arguments",
                cl.getClassLoader());

https://github.com/spring-projects/spring-hadoop/blob/2.0.4.RELEASE/spring-hadoop-core/src/main/java/org/springframework/data/hadoop/fs/DistCp.java#L274-L275

请注意，spring代码在Arguments类中查找Distcp内部类，但是这个内部类似乎不再存在于较新的Distcp代码中。当我运行这份工作时，我收到了这个错误

Caused by: java.lang.IllegalStateException: Cannot run distCp impersonated as 'null'
        at org.springframework.data.hadoop.fs.DistCp.copy(DistCp.java:268) ~[spring-data-hadoop-core-2.0.4.RELEASE.jar:2.0.4.RELEASE]
        at org.springframework.data.hadoop.fs.DistCp.copy(DistCp.java:216) ~[spring-data-hadoop-core-2.0.4.RELEASE.jar:2.0.4.RELEASE]
        at org.springframework.data.hadoop.fs.DistCp.copy(DistCp.java:152) ~[spring-data-hadoop-core-2.0.4.RELEASE.jar:2.0.4.RELEASE]
        at com.att.hadoop.hdfspub.source.hdfs.HdfsFileCopier.copyFolder(HdfsFileCopier.java:104) ~[classes/:na]
        ... 45 common frames omitted
Caused by: java.lang.IllegalArgumentException: Cannot find class [org.apache.hadoop.tools.DistCp$Arguments]
        at org.springframework.util.ClassUtils.resolveClassName(ClassUtils.java:286) ~[spring-core-4.1.4.RELEASE.jar:4.1.4.RELEASE]
        at org.springframework.data.hadoop.fs.DistCp.invokeCopy(DistCp.java:275) ~[spring-data-hadoop-core-2.0.4.RELEASE.jar:2.0.4.RELEASE]
        at org.springframework.data.hadoop.fs.DistCp.copy(DistCp.java:265) ~[spring-data-hadoop-core-2.0.4.RELEASE.jar:2.0.4.RELEASE]
        ... 48 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.tools.DistCp$Arguments

在我的pom.xml中，我包含了spring-data-hadoop 2.0.4.RELEASE和hadoop-distcp.2.2.0。

Answer 1

这已经在最近的2.1 RC1版本中得到解决，我们还没有考虑向2.0.x分支向后移植。如果您想尝试使用2.1.0.RC1版本，则需要在构建中包含Spring IO repo - 请参阅项目页面的“快速入门”部分 - http://projects.spring.io/spring-hadoop/

在hadoop 2.x中从spring运行distcp作业

1 个答案: