在org.apache.hadoop.util.ReflectionUtils.setJobConf中配置对象时出错

时间：2014-10-06 05:08:17

标签： java ubuntu hadoop mapreduce runtimeexception

我经历了很多关于此错误的问题，但找不到任何可以解决我的问题的解决方案。 在这里，我正在使用Hadoop实现对Twitter数据的情绪分析。

主类：

public class SentimentAnalysis extends Configured implements Tool{
private static File file;

public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();
    Classify classify = new Classify();

    /**
     * Mapper which reads Tweets text file Store 
     * as <"Positive",1> or <"Negative",1>
     */
    public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter)
            throws IOException {
        String line = value.toString();//streaming each tweet from the text file
        if (line != null) {
            word.set(classify.classify(line)); //invoke classify class to get tweet group of each text
            output.collect(word, one);
        } else {
            word.set("Error");
            output.collect(word, one);//Key,value for Mapper
        }
    }
}
public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
    /**
     * Count the frequency of each classified text group
     */

    @Override
    public void reduce(Text key, Iterator<IntWritable> classifiedText,
            OutputCollector<Text, IntWritable> output, Reporter reporter)
            throws IOException {
        int sum = 0;
        while (classifiedText.hasNext()) {
            sum += classifiedText.next().get(); //Sum the frequency
        }
        output.collect(key, new IntWritable(sum));
    }
}
public static class Classify {
    String[] categories;
    @SuppressWarnings("rawtypes")
    LMClassifier lmc;

    /**
     * Constructor loading serialized object created by Model class to local
     * LMClassifer of this class
     */
    @SuppressWarnings("rawtypes")
    public Classify() {
        try {

            lmc = (LMClassifier) AbstractExternalizable.readObject(file);
            categories = lmc.categories();
        } catch (ClassNotFoundException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    /**
     * Classify whether the text is positive or negative based on Model object
     * 
     * @param text
     * @return classified group i.e either positive or negative
     */
    public String classify(String text) {
        ConditionalClassification classification = lmc.classify(text);
        return classification.bestCategory();
    }
}

public static void main(String[] args) throws Exception {
    int ret = ToolRunner.run(new SentimentAnalysis(), args);
    System.exit(ret);
}

@Override
public int run(String[] args) throws Exception {
    if(args.length < 2) {
        System.out.println("Invalid input and output directories");
        return -1;
    }
    JobConf conf = new JobConf(getConf(), SentimentAnalysis.class);
    conf.setJobName("sentimentanalysis");
    conf.setJarByClass(SentimentAnalysis.class);
    conf.setOutputKeyClass(Text.class);
    conf.setOutputValueClass(IntWritable.class);
    conf.setMapOutputKeyClass(Text.class);
    conf.setMapOutputValueClass(IntWritable.class);
    conf.setMapperClass(Map.class);
    //conf.setCombinerClass(Reduce.class);
    conf.setReducerClass(Reduce.class);
    conf.setInputFormat(TextInputFormat.class);
    conf.setOutputFormat(TextOutputFormat.class);
    FileInputFormat.setInputPaths(conf, new Path(args[0]));
    FileOutputFormat.setOutputPath(conf, new Path(args[1]));
    file = new File(args[2]);
    JobClient.runJob(conf);
    return 0;
}
}

错误：

[cloudera@localhost ~]$ hadoop jar Sentiment.jar SentimentAnalysis test.txt SentimentOutput classifier.txt

Test.txt包含很少的推文需要分析其情绪。 Classifier.txt是一个编码文本文件，它有助于Classify（LMClassifier）类分析Test.txt中存在的推文。

14/10/05 20:59:23 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
14/10/05 20:59:24 INFO mapred.FileInputFormat: Total input paths to process : 1
14/10/05 20:59:24 INFO mapred.JobClient: Running job: job_201410041909_0035
14/10/05 20:59:25 INFO mapred.JobClient:  map 0% reduce 0%
14/10/05 20:59:41 INFO mapred.JobClient: Task Id : attempt_201410041909_0035_m_000000_0, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/10/05 20:59:41 INFO mapred.JobClient: Task Id : attempt_201410041909_0035_m_000001_0, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/10/05 20:59:52 INFO mapred.JobClient: Task Id : attempt_201410041909_0035_m_000000_1, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/10/05 20:59:53 INFO mapred.JobClient: Task Id : attempt_201410041909_0035_m_000001_1, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/10/05 21:00:04 INFO mapred.JobClient: Task Id : attempt_201410041909_0035_m_000000_2, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/10/05 21:00:04 INFO mapred.JobClient: Task Id : attempt_201410041909_0035_m_000001_2, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/10/05 21:00:19 INFO mapred.JobClient: Job complete: job_201410041909_0035
14/10/05 21:00:19 INFO mapred.JobClient: Counters: 7
14/10/05 21:00:19 INFO mapred.JobClient:   Job Counters 
14/10/05 21:00:19 INFO mapred.JobClient:     Failed map tasks=1
14/10/05 21:00:19 INFO mapred.JobClient:     Launched map tasks=8
14/10/05 21:00:19 INFO mapred.JobClient:     Data-local map tasks=8
14/10/05 21:00:19 INFO mapred.JobClient:     Total time spent by all maps in occupied slots (ms)=98236
14/10/05 21:00:19 INFO mapred.JobClient:     Total time spent by all reduces in occupied slots (ms)=0
14/10/05 21:00:19 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
14/10/05 21:00:19 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
14/10/05 21:00:19 INFO mapred.JobClient: Job Failed: NA
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1416)
at SentimentAnalysis.run(SentimentAnalysis.java:124)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at SentimentAnalysis.main(SentimentAnalysis.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

1 个答案:

答案 0 :(得分：1)

我个人没有使用Hadoop的经验，但是如果你＆＃34;看起来＆＃34;在堆栈跟踪中，它似乎是org.apache.hadoop.util.ReflectionUtils.setJobConf中的运行时异常... 

private static void setJobConf(Object theObject, Configuration conf) {
75     //If JobConf and JobConfigurable are in classpath, AND
76     //theObject is of type JobConfigurable AND
77     //conf is of type JobConf then
78     //invoke configure on theObject
79     try {
80       Class<?> jobConfClass = 
81         conf.getClassByName("org.apache.hadoop.mapred.JobConf");
82       Class<?> jobConfigurableClass = 
83         conf.getClassByName("org.apache.hadoop.mapred.JobConfigurable");
84        if (jobConfClass.isAssignableFrom(conf.getClass()) &&
85             jobConfigurableClass.isAssignableFrom(theObject.getClass())) {
86         Method configureMethod = 
87           jobConfigurableClass.getMethod("configure", jobConfClass);
88         configureMethod.invoke(theObject, conf);
89       }
90     } catch (ClassNotFoundException e) {
91       //JobConf/JobConfigurable not in classpath. no need to configure
92     } catch (Exception e) {
93       throw new RuntimeException("Error in configuring object", e);
94     }
95   }

显然，JobConf和JobConfigurable类都在类路径中（否则它会通过CNFE catch块掉落）所以发生了另一个异常......看起来好像嵌套异常是java.lang.reflect.InvocationTargetException表明调用＆＃39;存在问题。在上面第88行。

因此，尝试调用“配置”＆＃39;传入配置的目标作业实例上的方法失败。

InvocationTargetException可能包含了实际的因果异常，因此您需要以某种方式捕获顶层的RuntimeException，然后是e.getCause（）。getCause（）。printStackTrace（）以找出调用configure方法失败的原因。

相关问题
最新问题