无法使用Java / Java8向大型Zip添加条目

时间:2018-08-30 14:29:39

标签: java java-8

关于以下帖子: Appending files to a zip file with Java

我正在尝试解压缩大约5GB的巨大zip文件,因为我的瓶颈是I / O写入,所以我花费的最佳时间是大约30分钟,因此我尝试更改方法并在ZIP本身内部进行修改。 / p>

我的情况很简单:添加几个文件/文件/文件夹,该代码在小型zip文件上可以正常使用,但是在大型zip文件上却不起作用。

我尝试了很多组合...尝试了一些尝试:

第一:

   public static void main(String[] args) {
        String zip = "/Users/xxx/Documents/test/before.zip";

        Map<String, String> env = new HashMap<>();
        env.put("create", "true");
        Path path = Paths.get(zip);
        URI uri = URI.create("jar:" + path.toUri());
        try (FileSystem fs = FileSystems.newFileSystem(uri, env)) {
            Path toAdd = Paths.get("/Users/xxx/Documents/test/append.txt");
                Path directory = fs.getPath("/some/directory");

                //create if not exists
                Files.createDirectories(directory);

                //add to dir new file
                Files.copy(toAdd, directory.resolve("replaced.txt"));
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

但是我得到了

java.lang.OutOfMemoryError: Java heap space

at sun.util.calendar.Gregorian.newCalendarDate(Gregorian.java:85)
at sun.util.calendar.Gregorian.newCalendarDate(Gregorian.java:37)
at java.util.Date.<init>(Date.java:254)
at com.sun.nio.zipfs.ZipUtils.dosToJavaTime(ZipUtils.java:122)
at com.sun.nio.zipfs.ZipFileSystem$Entry.cen(ZipFileSystem.java:1884)
at com.sun.nio.zipfs.ZipFileSystem$Entry.readCEN(ZipFileSystem.java:1871)
at com.sun.nio.zipfs.ZipFileSystem.sync(ZipFileSystem.java:1257)
at com.sun.nio.zipfs.ZipFileSystem.close(ZipFileSystem.java:277)
at TrueZi.AAA.append200Files7KBExistingFolder(AAA.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.junit.internal.runners.TestMethodRunner.executeMethodBody(TestMethodRunner.java:99)
at org.junit.internal.runners.TestMethodRunner.runUnprotected(TestMethodRunner.java:81)
at org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34)
at org.junit.internal.runners.TestMethodRunner.runMethod(TestMethodRunner.java:75)
at org.junit.internal.runners.TestMethodRunner.run(TestMethodRunner.java:45)
at org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod(TestClassMethodsRunner.java:71)
at org.junit.internal.runners.TestClassMethodsRunner.run(TestClassMethodsRunner.java:35)
at org.junit.internal.runners.TestClassRunner$1.runUnprotected(TestClassRunner.java:42)
at org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34)
at org.junit.internal.runners.TestClassRunner.run(TestClassRunner.java:52)
at org.junit.runner.JUnitCore.run(JUnitCore.java:121)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

java.lang.OutOfMemoryError: Java heap space

    at com.sun.nio.zipfs.ZipFileSystem$Entry.readCEN(ZipFileSystem.java:1871)
    at com.sun.nio.zipfs.ZipFileSystem.sync(ZipFileSystem.java:1257)
    at com.sun.nio.zipfs.ZipFileSystem.close(ZipFileSystem.java:277)
    at TrueZi.AAA.test(AAA.java:84)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.junit.internal.runners.TestMethodRunner.executeMethodBody(TestMethodRunner.java:99)
    at org.junit.internal.runners.TestMethodRunner.runUnprotected(TestMethodRunner.java:81)
    at org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34)
    at org.junit.internal.runners.TestMethodRunner.runMethod(TestMethodRunner.java:75)
    at org.junit.internal.runners.TestMethodRunner.run(TestMethodRunner.java:45)
    at org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod(TestClassMethodsRunner.java:71)
    at org.junit.internal.runners.TestClassMethodsRunner.run(TestClassMethodsRunner.java:35)
    at org.junit.internal.runners.TestClassRunner$1.runUnprotected(TestClassRunner.java:42)
    at org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34)...

可能是这个吗?

https://community.oracle.com/thread/4123064

另一种说法是(将zip改写为其他zip名称并附加新文件,但平均而言,我需要花费与解压缩相同的时间):

ZipInputStream original = new ZipInputStream(new BufferedInputStream(new FileInputStream("/Users/xxx/Documents/test/original.zip")));
        ZipOutputStream append = new ZipOutputStream(new FileOutputStream("/Users/xxx/Documents/test/rename.zip"));
        ZipEntry entry;
        while ((entry = original.getNextEntry()) != null) {
            append.putNextEntry(entry);
            if (!entry.isDirectory()) {
                IOUtils.copy(original, append);
            }
            append.closeEntry();
        }

        File f = new File(appended);
        try (FileInputStream inputStream = new FileInputStream(f)) {
            append.putNextEntry(new ZipEntry(f.getName()));
            byte[] readBuffer = new byte[4096];
            int amountRead;

            while ((amountRead = inputStream.read(readBuffer)) > 0) {
                append.write(readBuffer, 0, amountRead);
            }

        } catch (Exception e) {
            e.printStackTrace();
        }


    // close
    original.close();
    append.close();

我想知道是否还有其他任何选择,既不使用FileSystem,又不将所有巨大的zip重写为其他。...??

1 个答案:

答案 0 :(得分:0)

请利用

env.put("useTempFile", Boolean.TRUE);

,如here所述。那么您将不会收到OutOfMemory错误