Question

我有一个java应用程序，它将json消息转换为镶木地板格式。是否有任何镶嵌写入器在java中写入缓冲区或字节流？我见过的大多数例子都只是文件。

Answer 1

TLDR；您将需要实现OutputFile，例如类似于：

import org.apache.parquet.io.OutputFile;
import org.apache.parquet.io.PositionOutputStream;

import java.io.BufferedOutputStream;
import java.io.IOException;

public class ParquetBufferedWriter implements OutputFile {

    private final BufferedOutputStream out;

    public ParquetBufferedWriter(BufferedOutputStream out) {
        this.out = out;
    }

    @Override
    public PositionOutputStream create(long blockSizeHint) throws IOException {
        return createPositionOutputstream();
    }

    private PositionOutputStream createPositionOutputstream() {
        return new PositionOutputStream() {
            @Override
            public long getPos() throws IOException {
                return 0;
            }

            @Override
            public void write(int b) throws IOException {
                out.write(b);
            }
        };
    }

    @Override
    public PositionOutputStream createOrOverwrite(long blockSizeHint) throws IOException {
        return createPositionOutputstream();
    }

    @Override
    public boolean supportsBlockSize() {
        return false;
    }

    @Override
    public long defaultBlockSize() {
        return 0;
    }

}

您的作家将是这样：

    ParquetBufferedWriter out = new ParquetBufferedWriter();
        try (ParquetWriter<Record> writer = AvroParquetWriter.
                <Record>builder(out)
                .withRowGroupSize(DEFAULT_BLOCK_SIZE)
                .withPageSize(DEFAULT_PAGE_SIZE)
                .withSchema(SCHEMA)
                .build()) {

            for (Record record : records) {
                writer.write(record);
            }
        } catch (IOException e) {
            throw new IllegalStateException(e);
        }

Answer 2

我还需要写入流，所以我完成了naimdjon给出的示例。以下对我来说非常合适。

lst

Answer 3

看看how they store a CSV file并尝试适应您的需求。

Answer 4

您需要将数据写入临时文件，然后将数据从文件隐蔽到输入流或缓冲区像这样，首先读取tempfile数据

final InputStream targetStream = new DataInputStream(new FileInputStream(tmp1.getAbsoluteFile()));

StringWriter writer = new StringWriter();
String encoding = StandardCharsets.UTF_8.name();
IOUtils.copy(targetStream, writer, encoding);
System.out.println(writer);

Parquet Writer缓冲区或字节流

4 个答案: