Question

我已经开始使用Apache Pig进行我们的一个项目。我必须创建一个自定义输入格式来加载我们的数据文件。为此，我遵循了这个例子Hadoop:Custom Input format。我还创建了自定义RecordReader实现来读取数据（我们从其他应用程序获取二进制格式的数据）并将其解析为正确的JSON格式。

在Pig脚本中使用自定义加载程序时会出现问题。一旦我的loader的getNext（）方法被调用，它就会调用我的自定义RecordReader的nextKeyValue（）方法，它可以正常工作。它正确读取数据，将其传递回我的加载器，后者解析数据并返回一个元组。到目前为止一切都很好。

当我的loader的getNext（）方法被一次又一次地调用时，问题出现了。它被调用，工作正常，并返回正确的输出（我调试它直到return语句）。但是，不是让执行更进一步，我的加载器再次被调用。我试着看看我的装载机被叫的次数，我可以看到这个号码直到20K！

有人可以帮我理解我的代码中的问题吗？

装载机

public class SimpleTextLoaderCustomFormat extends LoadFunc {

protected RecordReader in = null;
private byte fieldDel = '\t';
private ArrayList<Object> mProtoTuple = null;
private TupleFactory mTupleFactory = TupleFactory.getInstance();

@Override
public Tuple getNext() throws IOException {
    Tuple t = null;
    try {
        boolean notDone = in.nextKeyValue();
        if (!notDone) {
            return null;
        }
        String value = (String) in.getCurrentValue();
        byte[] buf = value.getBytes();
        int len = value.length();
        int start = 0;

        for (int i = 0; i < len; i++) {
            if (buf[i] == fieldDel) {
                readField(buf, start, i);
                start = i + 1;
            }
        }
        // pick up the last field
        readField(buf, start, len);

        t =  mTupleFactory.newTupleNoCopy(mProtoTuple);
        mProtoTuple = null;

    } catch (InterruptedException e) {
        int errCode = 6018;
        String errMsg = "Error while reading input";
        e.printStackTrace();
        throw new ExecException(errMsg, errCode,
                PigException.REMOTE_ENVIRONMENT, e);
    }
    return t;
}

private void readField(byte[] buf, int start, int end) {
    if (mProtoTuple == null) {
        mProtoTuple = new ArrayList<Object>();
    }

    if (start == end) {
        // NULL value
        mProtoTuple.add(null);
    } else {
        mProtoTuple.add(new DataByteArray(buf, start, end));
    }

}

@Override
public InputFormat getInputFormat() throws IOException {
    //return new TextInputFormat();
    return new CustomStringInputFormat();
}

@Override
public void setLocation(String location, Job job) throws IOException {
    FileInputFormat.setInputPaths(job, location);
}

@Override
public void prepareToRead(RecordReader reader, PigSplit split)
        throws IOException {
    in = reader;
}

自定义输入格式

public class CustomStringInputFormat extends FileInputFormat<String, String> {

    @Override
    public RecordReader<String, String> createRecordReader(InputSplit arg0,
            TaskAttemptContext arg1) throws IOException, InterruptedException {
        return new CustomStringInputRecordReader();
    }

}

自定义记录阅读器

public class CustomStringInputRecordReader extends RecordReader<String, String> {

    private String fileName = null;
    private String data = null;
    private Path file = null;
    private Configuration jc = null;
    private static int count = 0;

    @Override
    public void close() throws IOException {
//      jc = null;
//      file = null;
    }

    @Override
    public String getCurrentKey() throws IOException, InterruptedException {
        return fileName;
    }

    @Override
    public String getCurrentValue() throws IOException, InterruptedException {
        return data;
    }

    @Override
    public float getProgress() throws IOException, InterruptedException {
        return 0;
    }

    @Override
    public void initialize(InputSplit genericSplit, TaskAttemptContext context)
            throws IOException, InterruptedException {
        FileSplit split = (FileSplit) genericSplit;
        file = split.getPath();
        jc = context.getConfiguration();
    }

    @Override
    public boolean nextKeyValue() throws IOException, InterruptedException {
        InputStream is = FileSystem.get(jc).open(file);
        StringWriter writer = new StringWriter();
        IOUtils.copy(is, writer, "UTF-8");
        data = writer.toString();
        fileName = file.getName();
        writer.close();
        is.close();

        System.out.println("Count : " + ++count);

        return true;
    }

}

Answer 1

在装载程序中尝试此操作

// ...

boolean notDone = ((CustomStringInputFormat)in).nextKeyValue();

// ...

Text value = new Text(((CustomStringInputFormat))in.getCurrentValue().toString())

PIG自定义加载器的getNext（）被一次又一次地调用

1 个答案: