Question

我编写了一个map reduce作业来读取文件中的数据并将其插入Hbase表中。但我面临的问题是只有1条记录插入Hbase表中。我不确定这是最后一条记录还是任何随机记录，因为我的输入文件大约是10Gb。我写的逻辑，我确信记录应该插入表中的数千个。我只分享reducer代码和Driver类代码，因为我很确定，问题出在这里。请找到下面的代码：

public static class Reduce extends TableReducer<Text,Text,ImmutableBytesWritable> {


        public void reduce(Text key, Iterable<Text> values, Context context)
                throws IOException, InterruptedException {

            Set<Text> uniques = new HashSet<Text>();
            String vis=key.toString();
            String[] arr=vis.split(":");

            Put put=null;
            for (Text val : values){
                if (uniques.add(val)) {
                put = new Put(arr[0].getBytes());
                put.add(Bytes.toBytes("cf"), Bytes.toBytes("column"),Bytes.toBytes(val.toString()));

                }
                context.write(new ImmutableBytesWritable(arr[0].getBytes()), put); 
            }

        }
    }

我的驱动程序类：

        Configuration conf =  HBaseConfiguration.create();
        Job job = new Job(conf, "Blank");
        job.setJarByClass(Class_name.class);

        job.setMapperClass(Map.class);
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(Text.class);

        job.setSortComparatorClass(CompositeKeyComprator.class);

        Scan scan = new Scan();
        scan.setCaching(500);       
        scan.setCacheBlocks(false); 


        job.setReducerClass(Reduce.class);
        TableMapReduceUtil.initTableReducerJob(
                "Table_name",
                Reduce.class,
                job);           

        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));

        job.waitForCompletion(true);

在控制台中运行程序后，它表示Reduce输出记录= 73579，但在表中只插入了1条记录。

15/06/19 16:32:41 INFO mapred.JobClient: Job complete: job_201506181703_0020
15/06/19 16:32:41 INFO mapred.JobClient: Counters: 28
15/06/19 16:32:41 INFO mapred.JobClient:   Map-Reduce Framework
15/06/19 16:32:41 INFO mapred.JobClient:     Spilled Records=147158
15/06/19 16:32:41 INFO mapred.JobClient:     Map output materialized bytes=6941462
15/06/19 16:32:41 INFO mapred.JobClient:     Reduce input records=73579
15/06/19 16:32:41 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=7614308352
15/06/19 16:32:41 INFO mapred.JobClient:     Map input records=140543
15/06/19 16:32:41 INFO mapred.JobClient:     SPLIT_RAW_BYTES=417
15/06/19 16:32:41 INFO mapred.JobClient:     Map output bytes=6794286
15/06/19 16:32:41 INFO mapred.JobClient:     Reduce shuffle bytes=6941462
15/06/19 16:32:41 INFO mapred.JobClient:     Physical memory (bytes) snapshot=892702720
15/06/19 16:32:41 INFO mapred.JobClient:     Reduce input groups=1
15/06/19 16:32:41 INFO mapred.JobClient:     Combine output records=0
15/06/19 16:32:41 INFO mapred.JobClient:     Reduce output records=73579
15/06/19 16:32:41 INFO mapred.JobClient:     Map output records=73579
15/06/19 16:32:41 INFO mapred.JobClient:     Combine input records=0
15/06/19 16:32:41 INFO mapred.JobClient:     CPU time spent (ms)=10970
15/06/19 16:32:41 INFO mapred.JobClient:     Total committed heap usage (bytes)=829947904
15/06/19 16:32:41 INFO mapred.JobClient:   File Input Format Counters
15/06/19 16:32:41 INFO mapred.JobClient:     Bytes Read=204120920
15/06/19 16:32:41 INFO mapred.JobClient:   FileSystemCounters
15/06/19 16:32:41 INFO mapred.JobClient:     HDFS_BYTES_READ=204121337
15/06/19 16:32:41 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=14198205
15/06/19 16:32:41 INFO mapred.JobClient:     FILE_BYTES_READ=6941450
15/06/19 16:32:41 INFO mapred.JobClient:   Job Counters

当我将reducer输出写入文件时，我得到正确的输出。但不是在Hbase表中。请告诉我这里缺少的东西。提前谢谢。

Answer 1

您正在同一列系列和列限定符下使用相同的行键将数据插入HBase。根据您的计数器统计信息，您只有1个reducer组。因此，您的所有数据都会被覆盖在同一个单元格中。这就是你在HBase表中只获得一行的原因。

无法使用MapReduce将数据插入Hbase表

1 个答案: