如何实现hadoop的组比较器?

时间:2014-04-02 06:48:46

标签: java hadoop bigdata

给定一个名为KeyLabelDistance的类,我将其作为Hadoop中的键和值传递,我想对其执行二次排序,即我首先要根据键的增加值对键进行排序,然后按递减顺序排序距离。

为了做到这一点,我需要编写自己的GroupingComparator。我的问题是,因为setGroupingComparator()方法仅将一个扩展RawComparator的类作为参数,我如何在分组比较器中执行此比较字节?我是否需要显式序列化和反序列化对象? 并且如下所示让KeyLabelDistance类实现WritableComparable需要将SortComparator作为冗余吗?

我从这个答案中使用了SortComparator和GroupComparator:What are the differences between Sort Comparator and Group Comparator in Hadoop?

以下是KeyLabelDistance的实现:

public class KeyLabelDistance implements WritableComparable<KeyLabelDistance>
    {
        private int key;
        private int label;
        private double distance;
        KeyLabelDistance()
        {
            key = 0;
            label = 0;
            distance = 0;
        }
        KeyLabelDistance(int key, int label, double distance)
        {
            this.key = key;
            this.label = label;
            this.distance = distance;
        }
        public int getKey() {
            return key;
        }
        public void setKey(int key) {
            this.key = key;
        }
        public int getLabel() {
            return label;
        }
        public void setLabel(int label) {
            this.label = label;
        }
        public double getDistance() {
            return distance;
        }
        public void setDistance(double distance) {
            this.distance = distance;
        }

        public int compareTo(KeyLabelDistance lhs, KeyLabelDistance rhs)
        {
            if(lhs == rhs)
                return 0;
            else
            {
                if(lhs.getKey() < rhs.getKey())
                    return -1;
                else if(lhs.getKey() > rhs.getKey())
                    return 1;
                else
                {
                    //If the keys are equal, look at the distances -> since more is the "distance" more is the "similarity", the comparison is counterintuitive
                    if(lhs.getDistance() < rhs.getDistance() )
                        return 1;
                    else if(lhs.getDistance() > rhs.getDistance())
                        return -1;
                    else return 0;
                }
            }
        }
    }

组比较器的代码如下:

public class KeyLabelDistanceGroupingComparator extends WritableComparator{
    public int compare (KeyLabelDistance lhs, KeyLabelDistance rhs)
    {
        if(lhs == rhs)
            return 0;
        else
        {
            if(lhs.getKey() < rhs.getKey())
                return -1;
            else if(lhs.getKey() > rhs.getKey())
                return 1;
            return 0;
        }
    }
}

感谢任何帮助。谢谢。

1 个答案:

答案 0 :(得分:0)

您可以扩展WritableComparator,后者又实现RawComparator。你的排序和分组比较器将扩展WritableComparator。

如果你不提供这些比较器,hadoop将在内部结束使用compareTo作为你的关键字。