Question

我有一种情况，即mapper作为自定义类型的对象发出键。它有两个字段，一个intWritable ID和一个数据数组IntArrayWritable。实施如下。 `

import java.io.*;

import org.apache.hadoop.io.*;

public class PairDocIdPerm implements WritableComparable<PairDocIdPerm> {

    public PairDocIdPerm(){
        this.permId = new IntWritable(-1);
        this.SignaturePerm = new IntArrayWritable();
    }


public IntWritable getPermId() {
        return permId;
    }



    public void setPermId(IntWritable permId) {
        this.permId = permId;
    }



    public IntArrayWritable getSignaturePerm() {
        return SignaturePerm;
    }



    public void setSignaturePerm(IntArrayWritable signaturePerm) {
        SignaturePerm = signaturePerm;
    }

    private IntWritable permId;
    private IntArrayWritable SignaturePerm;

    public PairDocIdPerm(IntWritable permId,IntArrayWritable SignaturePerm) {
   this.permId = permId;
   this.SignaturePerm = SignaturePerm;
   }



    @Override
    public void write(DataOutput out) throws IOException {
    permId.write(out);
    SignaturePerm.write(out);
    }

   @Override
    public void readFields(DataInput in) throws IOException {
    permId.readFields(in);
    SignaturePerm.readFields(in);
    }

    @Override
    public int hashCode() { // same permId must go to same reducer. there fore just permId
     return permId.get();//.hashCode(); 
    }

    @Override
    public boolean equals(Object o) {
     if (o instanceof PairDocIdPerm) {
   PairDocIdPerm tp = (PairDocIdPerm) o;
   return permId.equals(tp.permId) && SignaturePerm.equals(tp.SignaturePerm);
     }
     return false;
    }

    @Override
    public String toString() {
     return permId + "\t" +SignaturePerm.toString(); 
    }

    @Override
    public int compareTo(PairDocIdPerm tp) {
     int cmp = permId.compareTo(tp.permId);
     Writable[] ar, other;
     ar = this.SignaturePerm.get();
     other = tp.SignaturePerm.get();

    if (cmp == 0) {
     for(int i=0;i<ar.length;i++){
         if(((IntWritable)ar[i]).get() == ((IntWritable)other[i]).get()){cmp= 0;continue;}
        else if(((IntWritable)ar[i]).get() < ((IntWritable)other[i]).get()){ return -1;}
         else if(((IntWritable)ar[i]).get() > ((IntWritable)other[i]).get()){return 1;}
     }   
     }

     return cmp;
     //return 1;
   }

   }`

我要求具有相同Id的密钥与compareTo方法中编码的排序顺序相同的reducer。但是当我使用它时，我的作业执行状态始终是map100％reduce 0％。减少永远不会完成。这个实现有什么问题吗？一般情况下，如果reducer状态始终为0％，则可能出现的问题是什么。

Answer 1

我认为这可能是read方法中可能的空指针异常：

   @Override
    public void readFields(DataInput in) throws IOException {
    permId.readFields(in);
    SignaturePerm.readFields(in);
    }

在这种情况下，

permId为null。所以你需要做的是：

IntWritable permId = new IntWritable();

在字段初始化程序中或在读取之前。

但是，您的代码阅读起来很糟糕。

使用自定义Object作为mapper发出的键

1 个答案: