Question

我在服务器中有一个大的加密文件（10GB +）。我需要将解密后的文件以小块的形式传输到客户端。当客户端请求一个字节块（比如18到45）时，我必须随机访问该文件，读取特定字节，解密并使用ServletResponseStream将其传输到客户端。

但是由于文件是加密的，我必须将文件读取为16字节的块才能正确解密。

因此，如果客户端请求从字节18到45，在服务器中我必须读取16字节块的倍数的文件。所以我必须从第16位到第48位随机访问该文件。然后解密它。解密后，我必须从第一个跳过2个字节，从最后一个跳过3个字节，以返回所请求的相应数据客户端块。

这是我想要做的事情

调整加密文件的开始和结束

long start = 15; // input from client
long end = 45; // input from client
long skipStart = 0; // need to skip for encrypted file
long skipEnd = 0;

// encrypted files, it must be access in blocks of 16 bytes
if(fileisEncrypted){
   skipStart = start % 16;  // skip 2 byte at start
   skipEnd = 16 - end % 16; // skip 3 byte at end
   start = start - skipStart; // start becomes 16
   end = end + skipEnd; // end becomes 48
}

从头到尾访问加密文件数据

try(final FileChannel channel = FileChannel.open(services.getPhysicalFile(datafile).toPath())){
    MappedByteBuffer mappedByteBuffer = channel.map(FileChannel.MapMode.READ_ONLY, start, end-start);

    // *** No idea how to convert MappedByteBuffer into input stream ***
    // InputStream is = (How do I get inputstream for byte 16 to 48 here?)

    // the medhod I used earlier to decrypt the all file atonce, now somehow I need the inputstream of specific range
    is = new FileEncryptionUtil().getCipherInputStream(is,
                        EncodeUtil.decodeSeedValue(encryptionKeyRef), AESCipher.DECRYPT_MODE);

    // transfering decrypted input stream to servlet response
    OutputStream outputStream = response.getOutputStream();
    // *** now for chunk transfer, here I also need to 
    //     skip 2 bytes at the start and 3 bytes from the end. 
    //     How to do it? ***/
    org.apache.commons.io.IOUtils.copy(is, outputStream)
}

我错过了上面给出的代码中的几个步骤。我知道我可以尝试逐字节读取，从第一个开始忽略2byte，从最后一个字节开始忽略3个字节。但我不确定它是否足够有效。此外，客户端可以请求从字节18到2048的大块，这需要读取和解密几乎2千兆字节的数据。我担心创建一个大字节数组会占用太多内存。

如何在不对服务器处理或内存施加太大压力的情况下有效地执行此操作？有什么想法吗？

Answer 1

由于您还没有指定您正在使用的密码模式，我假设您在点击模式下使用AES，因为它设计用于读取随机数据块大文件而不必完全解密它们。

使用AES-CTR，您可以通过解密代码流式传输文件，并在可用时立即将块发送回客户端。所以你只需要一些内存中AES块大小的数组，其余的都是从磁盘中读取的。您需要添加特殊逻辑以跳过第一个和最后一个块的一些字节（但您不需要将整个内容加载到内存中）。

在另一个SO问题中有一个如何执行此操作的示例（这只执行seek）：Seeking in AES-CTR-encrypted input。之后，您可以跳过前几个字节，读取到最后一个块并将其调整为客户端请求的字节数。

Answer 2

经过一段时间的研究。这就是我解决它的方式。首先，我创建了一个ByteBufferInputStream类。阅读MappedByteBuffer

public class ByteBufferInputStream extends InputStream {
    private ByteBuffer byteBuffer;

    public ByteBufferInputStream () {
    }

    /** Creates a stream with a new non-direct buffer of the specified size. The position and limit of the buffer is zero. */
    public ByteBufferInputStream (int bufferSize) {
        this(ByteBuffer.allocate(bufferSize));
        byteBuffer.flip();
    }

    /** Creates an uninitialized stream that cannot be used until {@link #setByteBuffer(ByteBuffer)} is called. */
    public ByteBufferInputStream (ByteBuffer byteBuffer) {
        this.byteBuffer = byteBuffer;
    }

    public ByteBuffer getByteBuffer () {
        return byteBuffer;
    }

    public void setByteBuffer (ByteBuffer byteBuffer) {
        this.byteBuffer = byteBuffer;
    }

    public int read () throws IOException {
        if (!byteBuffer.hasRemaining()) return -1;
        return byteBuffer.get();
    }

    public int read (byte[] bytes, int offset, int length) throws IOException {
        int count = Math.min(byteBuffer.remaining(), length);
        if (count == 0) return -1;
        byteBuffer.get(bytes, offset, count);
        return count;
    }

    public int available () throws IOException {
        return byteBuffer.remaining();
    }
}

然后通过扩展InputStream创建BlockInputStream类，这将允许跳过额外的字节并以16字节块的倍数读取内部输入流。

public class BlockInputStream extends InputStream {
    private final BufferedInputStream inputStream;
    private final long totalLength;
    private final long skip;
    private long read = 0;
    private byte[] buff = new byte[16];
    private ByteArrayInputStream blockInputStream;

    public BlockInputStream(InputStream inputStream, long skip, long length) throws IOException {
        this.inputStream = new BufferedInputStream(inputStream);
        this.skip = skip;
        this.totalLength = length + skip;
        if(skip > 0) {
            byte[] b = new byte[(int)skip];
            read(b);
            b = null;
        }
    }


    private int readBlock() throws IOException {
        int count = inputStream.read(buff);
        blockInputStream = new ByteArrayInputStream(buff);
        return count;
    }

    @Override
    public int read () throws IOException {
        byte[] b = new byte[1];
        read(b);
        return (int)b[1];
    }

    @Override
    public int read(byte[] b) throws IOException {
        return read(b, 0, b.length);
    }

    @Override
    public int read (byte[] bytes, int offset, int length) throws IOException {
        long remaining = totalLength - read;
        if(remaining < 1){
            return -1;
        }
        int bytesToRead = (int)Math.min(length, remaining);
        int n = 0;
        while(bytesToRead > 0){
            if(read % 16 == 0 && bytesToRead % 16 == 0){
                int count = inputStream.read(bytes, offset, bytesToRead);
                read += count;
                offset += count;
                bytesToRead -= count;
                n += count;
            } else {
                if(blockInputStream != null && blockInputStream.available() > 0) {
                    int len = Math.min(bytesToRead, blockInputStream.available());
                    int count = blockInputStream.read(bytes, offset, len);
                    read += count;
                    offset += count;
                    bytesToRead -= count;
                    n += count;
                } else {
                    readBlock();
                }
            }
        }
        return n;
    }

    @Override
    public int available () throws IOException {
        long remaining = totalLength - read;
        if(remaining < 1){
            return -1;
        }
        return inputStream.available();
    }

    @Override
    public long skip(long n) throws IOException {
        return inputStream.skip(n);
    }

    @Override
    public void close() throws IOException {
        inputStream.close();
    }

    @Override
    public synchronized void mark(int readlimit) {
        inputStream.mark(readlimit);
    }

    @Override
    public synchronized void reset() throws IOException {
        inputStream.reset();
    }

    @Override
    public boolean markSupported() {
        return inputStream.markSupported();
    }
}

这是我使用这两个类的最终工作实现

private RangeData getRangeData(RangeInfo r) throws IOException, GeneralSecurityException, CryptoException {

    // used for encrypted files
    long blockStart = r.getStart();
    long blockEnd = r.getEnd();
    long blockLength = blockEnd - blockStart + 1;

    // encrypted files, it must be access in blocks of 16 bytes
    if(datafile.isEncrypted()){
        blockStart -= blockStart % 16;
        blockEnd = blockEnd | 15; // nearest multiple of 16 for length n = ((n−1)|15)+1
        blockLength = blockEnd - blockStart + 1;
    }

    try ( final FileChannel channel = FileChannel.open(services.getPhysicalFile(datafile).toPath()) )
    {
        MappedByteBuffer mappedByteBuffer = channel.map(FileChannel.MapMode.READ_ONLY, blockStart, blockLength);
        InputStream inputStream = new ByteBufferInputStream(mappedByteBuffer);
        if(datafile.isEncrypted()) {
            String encryptionKeyRef = (String) settingsManager.getSetting(AppSetting.DEFAULT_ENCRYPTION_KEY);
            inputStream = new FileEncryptionUtil().getCipherInputStream(inputStream,
                    EncodeUtil.decodeSeedValue(encryptionKeyRef), AESCipher.DECRYPT_MODE);
            long skipStart = r.getStart() - blockStart;
            inputStream = new BlockInputStream(inputStream, skipStart, r.getLength()); // this will trim the data to n bytes at last
        }
        return new RangeData(r, inputStream);
    }
}

如何从java中的大型加密文件中有效地读取给定范围的字节块？

2 个答案: