Question

有人可以提供一个示例代码段，了解如何将文件附加到现有的序列文件中吗？

下面是我用来附加到现有序列文件outputfile的代码，但是在追加后读取序列文件时会抛出校验和错误：

问题打开校验和文件：/ Users / {homedirectory} / Desktop / Sample / SequenceFile / outputfile。忽略异常：java.io.EOFException

public class AppendSequenceFile {

    /**
     * @param args
     * @throws IOException
     * @throws IllegalAccessException
     * @throws InstantiationException
     */
    public static void main(String[] args) throws IOException,
            InstantiationException, IllegalAccessException {

        Configuration conf = new Configuration();

        FileSystem fs = FileSystem.get(conf);
        Path inputFile = new Path("/Users/{homedirectory}/Desktop/Sample/SequenceFile/sampleAppendTextFiles");
        Path sequenceFile = new Path("/Users/{homedirectory}/Desktop/Sample/SequenceFile/outputfile");
        FSDataInputStream inputStream;
        Text key = new Text();
        Text value = new Text();
        SequenceFile.Writer writer = SequenceFile.createWriter(fs, conf,
                sequenceFile, key.getClass(), value.getClass());
        FileStatus[] fStatus = fs.listStatus(inputFile);

        for (FileStatus fst : fStatus) {
            String str = "";
            System.out.println("Processing file : " + fst.getPath().getName() + " and the size is : " + fst.getPath().getName().length());
            inputStream = fs.open(fst.getPath());
            key.set(fst.getPath().getName());
            while(inputStream.available()>0) {
                str = str+inputStream.readLine();
            }
            value.set(str);
            writer.append(key, value);

        }
    }
}

序列文件阅读器：

public class SequenceFileReader{
    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        FileSystem fs = FileSystem.get(conf);
        Path path = new Path("/Users/{homedirectory}/Desktop/Sample/SequenceFile/outputfile");
        SequenceFile.Reader reader = null;
        try {
            reader = new SequenceFile.Reader(fs, path, conf);
            Text key = new Text();
            Text value = new Text();
            while (reader.next(key, value)) { System.out.println(key);
            System.out.println(value);
            }
        } finally {
            IOUtils.closeStream(reader);
        }
    }
}

提前致谢。

Answer 1

我自己没有这样做，但是浏览我发现的Hadoop API文档。

您可以使用此API创建Writer。请参阅SequenceFile

public static org.apache.hadoop.io.SequenceFile.Writer createWriter(FileContext fc,Configuration conf,Path name,Class keyClass,Class valClass,org.apache.hadoop.io.SequenceFile.CompressionType compressionType,CompressionCodec codec,org.apache.hadoop.io.SequenceFile.Metadata metadata,EnumSet<CreateFlag> createFlag,org.apache.hadoop.fs.Options.CreateOpts... opts) throws IOException

在此API中，CreateFlag可以帮助您指定＆＃34; APPEND ＆＃34;选项。

Answer 2

在使用sequencefile阅读器对象读取文件之前，请尝试关闭用于追加文件的sequencefile writer对象。

附加到现有序列文件

2 个答案: