Question

我正在尝试使用ByteArrayOutputStream编写一个大小在1kb到10GB之间的文件，但抛出以下异常。我正在使用jdk 6.请建议任何更好的高性能Api。我正在使用相同的网络框来读写。

Exception in thread "main" java.lang.OutOfMemoryError:   Requested array size exceeds VM limit
        at java.util.Arrays.copyOf(Unknown Source)
        at java.io.ByteArrayOutputStream.grow(Unknown Source)
        at java.io.ByteArrayOutputStream.ensureCapacity(Unknown Source)
        at java.io.ByteArrayOutputStream.write(Unknown Source)
        at java.io.OutputStream.write(Unknown Source)
        at

代码：

import java.io.BufferedOutputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;


public class PrepareFile {

    /**
     * @param args
     * @throws Exception 
     */
    public static void main(String[] args) throws Exception {
        // TODO Auto-generated method stub

        new PrepareFile().constructFile("f:\\hello","f:\\output",10000000);

    }

    //Writes a large file of 10 GB using input file data of small size by duplicating
    public void constructFile(String fileName, String outPath, int multiplier) throws Exception {
        BufferedOutputStream fos = null;
        FileInputStream fis = null;

        final File inputFile = new File(fileName);
        String path = inputFile.getParent();
        if (outPath != null && !outPath.isEmpty()) {
            path = outPath;
        }

        fis = new FileInputStream(fileName);

        try {



            // read the transactions in the input file.
            byte[] txnData = new byte[(int) inputFile.length()];
            fis.read(txnData);

            final File outFile = new File(path, "Myfile");
            fos = new BufferedOutputStream(new FileOutputStream(outFile));
            final ByteArrayOutputStream baos = new ByteArrayOutputStream();
            final ByteArrayOutputStream baos1 = new ByteArrayOutputStream();

            //multiplier if input file size is 1 KB and output file is 10 GB, then multiplier value is (1024*1024)

            for (long i = 1; i <= multiplier; i++) {

                if(i >=40000 && i % 40000==0){
                    System.out.println("i value now: "+i);
                    baos.writeTo(fos);
                    baos.reset();
                    //baos.write(txnData);
                }

                // write transactions
                baos.write(txnData);
                baos1.write(txnData); //Exception is coming at this line
            }

            int Padding = myCustomMethod(baos1.toByteArray());

            // write all out data to the output stream
            baos.writeTo(fos);

            baos.flush();
            baos1.flush();
        } catch(Exception e){
            e.printStackTrace();
        }finally {
            fos.close();
            fis.close();
        }

    }

    public int myCustomMethod(byte[] b){

        //Need complete bytes to prepare the file trailer
        return 0;
    }


}

Answer 1

由于大小为32位签名，因此ByteArrayOutputStream不能包含2 GB或更多的缓冲区。

如果你想要性能，我会逐步处理文件并避免使用如此庞大的内存副本，因为它们非常昂贵。

BTW我有一个库Chronicle Bytes，它支持大于2 GB的缓冲区，可以使用本机内存并映射到文件以避免使用堆，并且可以比主内存大。

但是，如果逐步处理数据，则不需要这么大的缓冲区。

我还建议您使用Java 8，因为它比Java 6（十年前发布）更好地执行64位操作

编辑根据您的代码，无需使用ByteArrayOutputStream，您可以逐步准备文件。

//Writes a large file of 10 GB using input file data of small size by duplicating
public void constructFile(String fileName, String outFileName, int multiplier) throws IOException {
    byte[] bytes;
    try (FileInputStream fis = new FileInputStream(fileName)) {
        bytes = new byte[fis.available()];
        fis.read(bytes);
    }

    try (FileOutputStream fos = new FileOutputStream(outFileName)) {
        for (int i = 0; i < multiplier; i++) {
            fos.write(bytes);
        }
    }

    // now process the file "outFileName"
    // how depends on what you are trying to do.
    // NOTE: It is entirely possible the file should be processed as it is written.
}

Answer 2

虽然极端，你可以制作一个 Super ByteArrayOutputStream，里面隐藏了几个 ByteArrayOutputStreams（下面的例子使用了其中的 3 个，最大容量为 6 GB）：

public class LargeByteArrayOutputOutputStream extends OutputStream {

    private DirectByteArrayOutputStream b1 = new DirectByteArrayOutputStream(Integer.MAX_VALUE -8);
    private DirectByteArrayOutputStream b2 = new DirectByteArrayOutputStream(Integer.MAX_VALUE -8);
    private DirectByteArrayOutputStream b3 = new DirectByteArrayOutputStream(Integer.MAX_VALUE -8);

    private long posWrite = 0;
    private long posRead = 0;
    @Override
    public void write(int b) throws IOException {
        if (posWrite < b1.getArray().length) {
            b1.write(b);
        } else if (posWrite < ((long)b1.getArray().length + (long)b2.getArray().length)) {
            b2.write(b);
        } else {
            b3.write(b);
        }
        
        posWrite++;

    }
    
    public long length() {
        return posWrite;
    }

    /** Propably you may want to read afterward */      
    public int read() throws IOException 
    {
        if (posRead > posWrite) {
            return (int)-1;
        } else {
            byte b = 0;
            if (posRead < b1.getArray().length) {
                b = b1.getArray()[(int)posRead];
            } else if (posRead < ((long)b1.getArray().length + (long)b2.getArray().length)) {
                b = b2.getArray()[(int)(posRead - b1.getArray().length)];
            } else {
                b = b3.getArray()[(int)(posRead - ((long)b1.getArray().length + (long)b2.getArray().length))];
            }
            
            posRead++;
            return b;
        }
    }       
}


public class DirectByteArrayOutputStream extends java.io.ByteArrayOutputStream {

 public DirectByteArrayOutputStream(int size) {
   super(size);
 }

 /**
 * Reference to the byte array that backs this buffer.
 */
 public byte[] getArray() {
   return buf;
 }       

 protected void finalize() throws Throwable
 {
  super.finalize();
 }
}

Java-使用Byteoutputstream

2 个答案: