Question

我在Hadoop 0.20.2版本上开发了一个带有缩减端连接的MapReduce。所有java类都编译没有任何错误。我的程序按原样运行。但是这个版本的hadoop是在测试虚拟机上。在我真正的集群上，我有另一个版本的hadoop（hadoop 2.00-cdh4.1.2）。当我想编译java-classes时，它不起作用。我得到100个错误如下：

JoinMapper.java:8: package org.apache.hadoop.filecache does not exist
import org.apache.hadoop.filecache.DistributedCache;
                                  ^
JoinMapper.java:9: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.Path;
                           ^
JoinMapper.java:10: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
                           ^
JoinMapper.java:11: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.*;
^
JoinMapper.java:12: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.FileInputFormat;
                               ^
JoinMapper.java:13: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.FileOutputFormat;
                               ^
JoinMapper.java:14: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.InputFormat;
                               ^
JoinMapper.java:15: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.JobClient;
                               ^
JoinMapper.java:16: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.JobConf;
                               ^
JoinMapper.java:17: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.MapReduceBase;
                               ^
JoinMapper.java:18: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.Mapper;
                               ^
JoinMapper.java:19: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.OutputCollector;
                               ^
JoinMapper.java:20: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.Reporter;
                               ^
JoinMapper.java:21: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.TextOutputFormat;
                               ^
JoinMapper.java:22: package org.apache.hadoop.mapreduce.lib.input does not exist
import org.apache.hadoop.mapreduce.lib.input.MultipleInputs;
                                            ^
JoinMapper.java:23: package org.apache.hadoop.util does not exist
import org.apache.hadoop.util.Tool;
                             ^
JoinMapper.java:24: package org.apache.hadoop.util does not exist
import org.apache.hadoop.util.ToolRunner;
                             ^
JoinMapper.java:26: cannot find symbol
symbol: class MapReduceBase
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                ^
JoinMapper.java:26: cannot find symbol
symbol: class Mapper
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                                         ^
JoinMapper.java:26: cannot find symbol
symbol: class LongWritable
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                                                ^
JoinMapper.java:26: cannot find symbol
symbol: class Text
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                                                              ^
TextPair.java:2: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.*;
^
TextPair.java:4: cannot find symbol
symbol: class WritableComparable
public class TextPair implements WritableComparable<TextPair> {
                                 ^
TextPair.java:4: interface expected here
public class TextPair implements WritableComparable<TextPair> {
                                                   ^
JoinMapper.java:26: cannot find symbol
symbol: class Text
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                                                                              ^
JoinMapper.java:26: interface expected here
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                                               ^
NcdcRecordParser.java:2: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.*;

我知道这个版本没有我需要的软件包。但我不知道如何解决我的问题。任何人都可以帮助我吗？

Answer 1

据我所知，这些java包应该仍然存在于最新版本的Hadoop中。

我认为你的问题是hadoop jar不在你的类路径中。当我尝试编译没有它们的map reduce项目时，我遇到了类似的错误。

如果你正在使用javac，请尝试这样的事情（这使用我的Ubuntu CDH4开发机器上的路径位置）：

javac -classpath /usr/lib/hadoop/*:/usr/lib/hadoop-0.20-mapreduce/*:/usr/lib/hadoop-hdfs/* JoinMapper.java TextPair.java NcdcRecordParser.java

Answer 2

Apache Hadoop对其内部软件包进行了重新命名，介于0.20和0.23之间（也称为2.0.0）。这包括将org.apache.hadoop.mapred重命名为org.apache.hadoop.mapreduce。

Hadoop的Cloudera发行版具有单独的Maven工件，您可以依赖它们来构建旧版本。工件的名称列在CDH4 Maven page上;你想要版本为2.0.0-mr1-cdh4.1.2的工件。

另请参阅Is it better to use the mapred or the mapreduce package to create a Hadoop Job?，了解有关是否要移植到较新的API的一些详细信息。

Answer 3

为maven尝试param -Dhadoop.profile = 23。另请参阅https://issues.apache.org/jira/browse/HBASE-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098596#comment-13098596

hadoop版本的不兼容性

3 个答案: