Question

我对Hadoop和Hbase还是陌生的，我正在努力使它们一起工作。我建立了.java，并获得了.jar档案，没有任何错误。无论如何，在启动程序时，我会收到此错误：

    java.lang.Exception: java.io.IOException: wrong value class: class org.apache.hadoop.hbase.client.Put is not class org.apache.hadoop.io.IntWritable
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
    Caused by: java.io.IOException: wrong value class: class org.apache.hadoop.hbase.client.Put is not class org.apache.hadoop.io.IntWritable
at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:194)
at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:1378)
at org.apache.hadoop.mapred.Task$NewCombinerRunner$OutputConverter.write(Task.java:1695)
at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
at TweetSentiment$ClassificationCounterReducer.reduce(TweetSentiment.java:131)
at TweetSentiment$ClassificationCounterReducer.reduce(TweetSentiment.java:114)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1716)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1637)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1489)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:844)

我不明白为什么。

我已经尝试了一些不同版本的Hadoop和Hbase，但是它似乎不起作用。这是我的代码：

public static class ClassificationCounterReducer extends TableReducer<IntWritable,IntWritable,IntWritable> {

        //private IntWritable result = new IntWritable();

        public void reduce(IntWritable classification, Iterable<IntWritable> values, Context context)
                throws IOException, InterruptedException {
            int sum = 0;
            for (IntWritable val : values) {
                sum += val.get();
            }

            Put put = new Put(Bytes.toBytes(classification.toString()));

            put.addColumn( Bytes.toBytes("number"), Bytes.toBytes(""), Bytes.toBytes(sum) );
            context.write(classification, put);
        }
    }

    private static final String OUTPUT_TABLE = "sentiment";

    public int run(String[] args) throws Exception  {

        Job job = Job.getInstance(getConf(), "Sentiment Count");

        job.setJarByClass(TweetSentiment.class);
        job.setMapperClass(ClassificatorMapper.class);
        job.setCombinerClass(ClassificationCounterReducer.class);
        job.setReducerClass(ClassificationCounterReducer.class);

        TableMapReduceUtil.initTableReducerJob(
                OUTPUT_TABLE,
                TweetSentiment.ClassificationCounterReducer.class,
                job);

        job.setMapOutputKeyClass(IntWritable.class);
        job.setMapOutputValueClass(IntWritable.class);

        FileInputFormat.addInputPath(job, new Path(args[0]));

        System.exit(job.waitForCompletion(true) ? 0 : 1);

        return 0;
    }

    public static void main(String[] args) throws Exception {
        int res = ToolRunner.run(new HBaseConfiguration(), new TweetSentiment(), args);
        System.exit(res);
    }

该代码有望在Twitter的推文上进行情感分析。映射器应从HDFS中读取这些推文，对其进行分类，然后将结果发送到组合器/归约器。还原器应该先计数，然后将它们存储在hbase表中，该表具有两个条目，分别称为“情感”和“数字”。

Answer 1

您的类型不匹配-您正在输出String xPathString = String.Format("//img[@src='{0}']", link); item = WebDriverWait(driver, 100000).until(EC.presence_of_element_located((By.XPATH, xPathString)))，但是reducer类扩展了context.write(IntWritable, Put)。

将第一行更改为：

TableReducer<IntWritable,IntWritable,IntWritable>

此外，请取消设置组合器，它将不起作用。

java.lang.Exception：java.io.IOException：设置hadoop和hbase时值类错误

1 个答案: