Question

在MapReduce应用程序中，我有一个名为AnonymousPair的任意WritableComparable实现，我注意到了这个

import com.google.common.collect.MinMaxPriorityQueue;

public static class MyReducer extends Reducer<LongWritable, AnonymousPair, LongWritable, Text> {
    @Override
    protected void reduce(LongWritable key, Iterable<AnonymousPair> values, Context context) throws IOException, InterruptedException { 
        // ...
        MinMaxPriorityQueue<AnonymousPair> pQueue = MinMaxPriorityQueue
                .orderedBy(new AnonymousPair().comparator())
                .maximumSize(Constants.MaxKeywords)
                .create();

        for(AnonymousPair val : values) {
            pQueue.add(new AnonymousPair(val)); // No problem with copy constructor
            // pQueue.add(val);                 // Wrong! Every element in pQueue will be the same 
        }
    }
}

如果我不使用复制构造函数＆＃39;，pQueue中的每个元素最终都是相同的。谁能帮我理解这个？谢谢！我的猜测是

引用要修改的Reducer Values元素。它在文档的某个地方，但我错过了它。
我错误地使用Google Guava MinMaxPriorityQueue
我的WritableComparable实现有问题

我的AnonymousPair实施

    public static class AnonymousPair implements WritableComparable<AnonymousPair> {
        private String a = "";
        private Float b = 0f;
        public AnonymousPair() {}
        public AnonymousPair(String a, Float b) {this.a = a; this.b = b;}
        public AnonymousPair(AnonymousPair o) {this.a = o.a; this.b = o.b;}

        public Comparator<AnonymousPair> comparator() {return new AnonymousPairComparator();}

        class AnonymousPairComparator implements Comparator<AnonymousPair> {
            @Override
            public int compare(AnonymousPair o1, AnonymousPair o2) {
                Float diff = o1.b - o2.b;
                if(diff == 0) {
                    return 0;
                }
                if(diff < 0) {
                    return 1;   // Reverse order
                } else {
                    return -1;
                }

            }
        }

        @Override
        public int compareTo(AnonymousPair o) {
            int temp = this.a.compareTo(o.a);
            if(temp == 0) {
                return -this.b.compareTo(o.b);
            } else {
                return temp;
            }
        }

        // More overriding...
    }

Answer 1

请参阅javadoc：

框架将重用传递的键和值对象进入reduce，因此应用程序应克隆对象他们想保留一份副本。在许多情况下，所有值都合并在一起分为零或一个值。

需要复制MapReduce中的Reducer值，否则需要修改？

1 个答案: