Question

我有一个自定义（Mahout）类，我试图通过Hadoop运行。我已按照here的说明（以及将jar放入HDFS的一条评论）但我一直收到以下错误

ClassNotFoundException: org.apache.mahout.common.AbstractJob

我的设置如下：

    REP_ROOT=/home/user/.m2/repository
    MAHOUT_CORE=$REP_ROOT/org/apache/mahout/mahout-core/0.9/mahout-core-0.9.jar
    MAHOUT_CLI=$REP_ROOT/org/apache/mahout/commons/commons-cli/2.0-mahout/commons-cli-2.0-mahout.jar
    MAHOUT_INTEGRATION=$REP_ROOT/org/apache/mahout/mahout-integration/0.9/mahout-integration-0.9.jar

    LANG3=$REP_ROOT/org/apache/commons/commons-lang3/3.1/commons-lang3-3.1.jar
    GOOGLE_LIST=$REP_ROOT/com/google/guava/guava/14.0.1/guava-14.0.1.jar

    export LIBJARS=$MAHOUT_CORE,$MAHOUT_CLI,$MAHOUT_INTEGRATION,$LANG3,$GOOGLE_LIST
    export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$MAHOUT_CORE:$MAHOUT_CLI:$MAHOUT_INTEGRATION:$LANG3:$GOOGLE_LIST

    hadoop jar /home/user/bin/CustomJar.jar com.company.Mahout.LDAJob \
   -libjars ${LIBJARS} \
    -DbaseLocation=/user/hadoop/test \
    -Dinput=input \
   -Doutput=output

缺少的类（AbstractJob）包含在MAHOUT_CORE中。 LDAJob以：

开头

public class LDAJob extends AbstractJob{

public static void main(String args[]) throws Exception {
    Configuration conf = new Configuration();
    ToolRunner.run(conf,new LDAJob(), args);
}

public int run(String[] args) throws Exception {    
    Configuration conf = super.getConf();  
    String baseFileLocation = conf.get("baseLocation");
.....
}

错误详情

    14/02/26 01:29:44 INFO util.NativeCodeLoader: Loaded the native-hadoop library
14/02/26 01:29:44 WARN snappy.LoadSnappy: Snappy native library not loaded
14/02/26 01:29:47 INFO mapred.JobClient: Running job: job_201402260118_0004
14/02/26 01:29:48 INFO mapred.JobClient:  map 0% reduce 0%
14/02/26 01:29:53 INFO mapred.JobClient: Task Id : attempt_201402260118_0004_m_000000_0, Status : FAILED
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader.initNextRecordReader(CombineFileRecordReader.java:164)
    at org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader.<init>(CombineFileRecordReader.java:126)
    at org.apache.mahout.text.MultipleTextFileInputFormat.createRecordReader(MultipleTextFileInputFormat.java:43)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:488)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader.initNextRecordReader(CombineFileRecordReader.java:155)
    ... 10 more
Caused by: java.lang.NoClassDefFoundError: org/apache/mahout/common/AbstractJob
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
    at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at org.apache.mahout.text.WholeFileRecordReader.<init>(WholeFileRecordReader.java:61)
    ... 15 more
Caused by: java.lang.ClassNotFoundException: org.apache.mahout.common.AbstractJob
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    ... 28 more

非常感谢任何帮助。

在Hadoop 1.2.1中包含第三方JAR

0 个答案: