Question

我一直在单个节点上测试map reduce作业，它似乎工作但是现在我试图在远程集群上运行它我得到了ClassNotFoundExcepton。我的代码结构如下：

public class Pivot {
    public static class Mapper extends TableMapper<ImmutableBytesWritable, ImmutableBytesWritable> {
        @Override
        public void map(ImmutableBytesWritable rowkey, Result values, Context context) throws IOException {
            (map code)
        }
    }

    public static class Reducer extends TableReducer<ImmutableBytesWritable, ImmutableBytesWritable, ImmutableBytesWritable> {
        public void reduce(ImmutableBytesWritable key, Iterable<ImmutableBytesWritable> values, Context context) throws IOException, InterruptedException {
            (reduce code)
        }
    }

    public static void main(String[] args) {
        Configuration conf = HBaseConfiguration.create();
        conf.set("fs.default.name", "hdfs://hadoop-master:9000");
        conf.set("mapred.job.tracker", "hdfs://hadoop-master:9001");
        conf.set("hbase.master", "hadoop-master:60000");
        conf.set("hbase.zookeeper.quorum", "hadoop-master");
        conf.set("hbase.zookeeper.property.clientPort", "2222");
        Job job = new Job(conf);
        job.setJobName("Pivot");
        job.setJarByClass(Pivot.class);
        Scan scan = new Scan();
        TableMapReduceUtil.initTableMapperJob("InputTable", scan, Mapper.class, ImmutableBytesWritable.class, ImmutableBytesWritable.class, job);
        TableMapReduceUtil.initTableReducerJob("OutputTable", Reducer.class, job);
        job.waitForCompletion(true);
    }
}

我尝试运行此作业时收到的错误如下：

java.lang.RuntimeException: java.lang.ClassNotFoundException: Pivot$Mapper
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857)
    ...

有什么我想念的吗？为什么工作难以找到映射器？

Answer 1

从Eclipse运行作业时，重要的是要注意Hadoop要求您从jar启动作业。 Hadoop需要这样才能将代码发送到HDFS / JobTracker。

在你的情况下，我想你没有将你的工作类捆绑到一个罐子里，然后从罐子里运行程序 - 导致CNFE。

尝试构建一个jar并使用hadoop jar myjar.jar ...从命令行运行，一旦这个工作，那么你可以测试从Eclipse中运行

运行HBase map时ClassNotFoundException减少集群上的作业

1 个答案: