Question

我需要在Hbase上使用MR来比较hbase中的2个表table1，table2）并将汇总写入第三个表（table3）

我使用下面的TableMapReduceUtil伪代码。映射器：表1 减速机：表3.

在mapper中，我需要将Table1值与Table2进行比较。我在哪里实例化Table2？

在映射器中，我是否必须为每个映射器实例化Table3？我想实例化Table3 整个MapReduce工作只有一次？

driver()
{
TableMapReduceUtil.initTableMapperJob(
    table1,        // input table
    scan,              
    MyMapper.class,     // mapper class
    Text.class,         
    IntWritable.class,  
    job);
TableMapReduceUtil.initTableReducerJob(
    table3,        // output table
    MyTableReducer.class,    
    job);
}


public static class MyMapper extends TableMapper<Text, IntWritable>  {
    public static final byte[] CF = "cf".getBytes();
    public static final byte[] ATTR1 = "attr1".getBytes();

    private final IntWritable ONE = new IntWritable(1);
    private Text text = new Text();

    public void map(ImmutableBytesWritable row, Result value, Context context) throws IOException, InterruptedException {
            String val = new String(value.getValue(CF, ATTR1));
            String diff;
            //instantiate Table3 and compare with val. Do i have to instantiate for each mapper?

            text.set(diff);     

            context.write(text, ONE);
    }
}


public static class MyTableReducer extends TableReducer<Text, IntWritable, ImmutableBytesWritable>  {
    public static final byte[] CF = "cf".getBytes();
    public static final byte[] COUNT = "count".getBytes();

    public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
            int i = 0;
            for (IntWritable val : values) {
                i += val.get();
            }
            Put put = new Put(Bytes.toBytes(key.toString()));
            put.add(CF, COUNT, Bytes.toBytes(i));

            context.write(null, put);
    }
}

Answer 1

如果您尝试在HBase中创建一个已经创建的表，它将抛出一个您可以选择忽略的TableExistsException。见HBaseAdmin docs。所以你没问题 - 创建表的第一个映射器实际上会创建它，然后其他映射器将抛出你将忽略的异常。

比较hbase中的2个表，并使用TableMapReduceUtil将汇总写入第三个表

1 个答案: