如何将一个巨大的zip文件拆分成多个卷?

时间:2008-10-28 16:42:41

标签: java zip

当我通过java.util.zip.*创建zip存档时,有没有办法在多个卷中拆分生成的存档?

假设我的整体存档有filesize 24 MB,我想将其拆分为3个文件,每个文件的限制为10 MB。
是否有具有此功能的zip API?或者其他任何好方法来实现这个目标?

由于 Thollsten

5 个答案:

答案 0 :(得分:8)

检查:http://saloon.javaranch.com/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic&f=38&t=004618

我不知道任何可以帮助您做到这一点的公共API。 (虽然如果你不想以编程方式进行,可以使用像WinSplitter这样的实用程序)

我还没有尝试过,但是使用ZippedInput / OutputStream时每个ZipEntry都有一个压缩的大小。在创建压缩文件时,您可能会粗略估计压缩文件的大小。如果你需要2MB的压缩文件,那么你可以在累积的条目大小达到1.9MB之后停止写入文件,对于清单文件和其他特定于zip文件的元素需要.1MB。 因此,简而言之,您可以在ZippedInputStream上编写一个包装器,如下所示:

import java.util.zip.ZipOutputStream;
import java.util.zip.ZipEntry;
import java.io.FileOutputStream;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;

public class ChunkedZippedOutputStream {

    private ZipOutputStream zipOutputStream;

    private String path;
    private String name;

    private long currentSize;
    private int currentChunkIndex;
    private final long MAX_FILE_SIZE = 16000000; // Whatever size you want
    private final String PART_POSTFIX = ".part.";
    private final String FILE_EXTENSION = ".zip";

    public ChunkedZippedOutputStream(String path, String name) throws FileNotFoundException {
        this.path = path;
        this.name = name;
        constructNewStream();
    }

    public void addEntry(ZipEntry entry) throws IOException {
        long entrySize = entry.getCompressedSize();
        if((currentSize + entrySize) > MAX_FILE_SIZE) {
            closeStream();
            constructNewStream();
        } else {
            currentSize += entrySize;
            zipOutputStream.putNextEntry(entry);
        }
    }

    private void closeStream() throws IOException {
        zipOutputStream.close();
    }

    private void constructNewStream() throws FileNotFoundException {
        zipOutputStream = new ZipOutputStream(new FileOutputStream(new File(path, constructCurrentPartName())));
        currentChunkIndex++;
        currentSize = 0;
    }

    private String constructCurrentPartName() {
        // This will give names is the form of <file_name>.part.0.zip, <file_name>.part.1.zip, etc.
        StringBuilder partNameBuilder = new StringBuilder(name);
        partNameBuilder.append(PART_POSTFIX);
        partNameBuilder.append(currentChunkIndex);
        partNameBuilder.append(FILE_EXTENSION);
        return partNameBuilder.toString();
    }
}

上述程序只是一种方法的暗示,而不是任何方式的最终解决方案

答案 1 :(得分:5)

如果目标是让输出与pkzip和winzip兼容,我不知道有任何开源库可以做到这一点。我们对其中一个应用程序有类似的要求,最后编写了我们自己的实现(与zip标准兼容)。如果我记得,对我们来说最困难的事情是我们必须动态生成单个文件(大多数zip实用程序的工作方式是创建大型zip文件,然后返回并稍后拆分 - 这样更容易实现。大约需要一天的时间来编写,还有两天的时间来调试。

zip标准解释了文件格式的外观。如果你不怕卷起袖子,这绝对可行。您必须自己实现一个zip文件生成器,但是您可以使用Java的Deflator类为压缩数据生成段流。你必须自己生成文件和部分标题,但它们只是字节 - 一旦你潜入就没什么难的。

这是zip specification - 部分K具有您正在寻找的信息,但您还需要阅读A,B,C和F.如果你正在处理非常大的文件(我们是),你也必须进入Zip64的东西 - 但对于24 MB,你没问题。

如果你想潜入并尝试 - 如果你遇到问题,请回复,我会看看是否可以提供一些指示。

答案 2 :(得分:0)

下面的代码是我的解决方案,用于根据所需大小将目录结构中的zip文件拆分为大块。我发现以前的答案很有用,因此希望以相似但更简洁的方法做出贡献。此代码适合我的特定需求,并且我认为还有改进的余地。

private final static long MAX_FILE_SIZE = 1000 * 1000 * 1024; //  around 1GB 
private final static String zipCopyDest =  "C:\\zip2split\\copy";

public static void splitZip(String zipFileName, String zippedPath, String coreId) throws IOException{

    System.out.println("process whole zip file..");
    FileInputStream fis  = new FileInputStream(zippedPath);
    ZipInputStream zipInputStream = new ZipInputStream(fis);
    ZipEntry entry = null;
    int currentChunkIndex = 0;
    //using just to get the uncompressed size of the zipentries
    long entrySize = 0;
    ZipFile zipFile = new ZipFile(zippedPath);
    Enumeration enumeration = zipFile.entries();

    String copDest = zipCopyDest + "\\" + coreId + "_" + currentChunkIndex +".zip";

    FileOutputStream fos = new FileOutputStream(new File(copDest));
    BufferedOutputStream bos = new BufferedOutputStream(fos);
    ZipOutputStream zos = new ZipOutputStream(bos);
    long currentSize = 0; 

    try {
        while ((entry = zipInputStream.getNextEntry()) != null && enumeration.hasMoreElements()) {

            ZipEntry zipEntry = (ZipEntry) enumeration.nextElement();
            System.out.println(zipEntry.getName());
            System.out.println(zipEntry.getSize());
            entrySize = zipEntry.getSize();

            ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
            //long entrySize = entry.getCompressedSize();
            //entrySize = entry.getSize(); //gives -1

            if((currentSize + entrySize) > MAX_FILE_SIZE) {
                zos.close();
                //construct a new stream
                //zos = new ZipOutputStream(new FileOutputStream(new File(zippedPath, constructCurrentPartName(coreId))));
                currentChunkIndex++;
                zos = getOutputStream(currentChunkIndex, coreId);
                currentSize = 0;

            }else{
                currentSize += entrySize;
                zos.putNextEntry(new ZipEntry(entry.getName()));
                byte[] buffer = new byte[8192];
                int length = 0;
                while ((length = zipInputStream.read(buffer)) > 0) {
                    outputStream.write(buffer, 0, length);
                }

                byte[] unzippedFile = outputStream.toByteArray();
                zos.write(unzippedFile);
                unzippedFile = null;
                outputStream.close();
                zos.closeEntry();
            }
            //zos.close();
        }
    } finally {
        zos.close();
    }


}

 public static ZipOutputStream getOutputStream(int i, String coreId) throws IOException {
     System.out.println("inside of getOutputStream()..");
     ZipOutputStream out = new ZipOutputStream(new FileOutputStream(zipCopyDest + "\\" + coreId + "_" +  i +".zip"));   
    // out.setLevel(Deflater.DEFAULT_COMPRESSION);
     return out;
 }

public static void main(String args[]) throws IOException{
    String zipFileName = "Large_files _for_testing.zip";
    String zippedPath= "C:\\zip2split\\Large_files _for_testing.zip";
    String coreId = "Large_files _for_testing";
    splitZip(zipFileName, zippedPath, coreId);
}

答案 3 :(得分:0)

对于它的价值,我喜欢在任何地方使用 try-with-resources 。如果您喜欢那种设计模式,那么您会喜欢上这种方式。同样,如果条目大于所需的零件尺寸,这也解决了零件空的问题。在最坏的情况下,您至少要有 个部分。

在:

  

my-archive.zip

出局:

  

my-archive.part1of3.zip
  my-archive.part2of3.zip
  my-archive.part3of3.zip

注意:我正在使用日志记录和Apache Commons FilenameUtils,但是请随意使用工具箱中的内容。

/**
 * Utility class to split a zip archive into parts (not volumes)
 * by attempting to fit as many entries into a single part before
 * creating a new part. If a part would otherwise be empty because
 * the next entry won't fit, it will be added anyway to avoid empty parts.
 *
 * @author Eric Draken, 2019
 */
public class Zip
{
    private static final int DEFAULT_BUFFER_SIZE = 1024 * 4;

    private static final String ZIP_PART_FORMAT = "%s.part%dof%d.zip";

    private static final String EXT = "zip";

    private static final Logger logger = LoggerFactory.getLogger( MethodHandles.lookup().lookupClass() );

    /**
     * Split a large archive into smaller parts
     *
     * @param zipFile             Source zip file to split (must end with .zip)
     * @param outZipFile          Destination zip file base path. The "part" number will be added automatically
     * @param approxPartSizeBytes Approximate part size
     * @throws IOException Exceptions on file access
     */
    public static void splitZipArchive(
        @NotNull final File zipFile,
        @NotNull final File outZipFile,
        final long approxPartSizeBytes ) throws IOException
    {
        String basename = FilenameUtils.getBaseName( outZipFile.getName() );
        Path basePath = outZipFile.getParentFile() != null ? // Check if this file has a parent folder
            outZipFile.getParentFile().toPath() :
            Paths.get( "" );
        String extension = FilenameUtils.getExtension( zipFile.getName() );
        if ( !extension.equals( EXT ) )
        {
            throw new IllegalArgumentException( "The archive to split must end with ." + EXT );
        }

        // Get a list of entries in the archive
        try ( ZipFile zf = new ZipFile( zipFile ) )
        {
            // Silliness check
            long minRequiredSize = zipFile.length() / 100;
            if ( minRequiredSize > approxPartSizeBytes )
            {
                throw new IllegalArgumentException(
                    "Please select a minimum part size over " + minRequiredSize + " bytes, " +
                        "otherwise there will be over 100 parts."
                );
            }

            // Loop over all the entries in the large archive
            // to calculate the number of parts required
            Enumeration<? extends ZipEntry> enumeration = zf.entries();
            long partSize = 0;
            long totalParts = 1;
            while ( enumeration.hasMoreElements() )
            {
                long nextSize = enumeration.nextElement().getCompressedSize();
                if ( partSize + nextSize > approxPartSizeBytes )
                {
                    partSize = 0;
                    totalParts++;
                }
                partSize += nextSize;
            }

            // Silliness check: if there are more parts than there
            // are entries, then one entry will occupy one part by contract
            totalParts = Math.min( totalParts, zf.size() );

            logger.debug( "Split requires {} parts", totalParts );
            if ( totalParts == 1 )
            {
                // No splitting required. Copy file
                Path outFile = basePath.resolve(
                    String.format( ZIP_PART_FORMAT, basename, 1, 1 )
                );
                Files.copy( zipFile.toPath(), outFile );
                logger.debug( "Copied {} to {} (pass-though)", zipFile.toString(), outFile.toString() );
                return;
            }

            // Reset
            enumeration = zf.entries();

            // Split into parts
            int currPart = 1;
            ZipEntry overflowZipEntry = null;
            while ( overflowZipEntry != null || enumeration.hasMoreElements() )
            {
                Path outFilePart = basePath.resolve(
                    String.format( ZIP_PART_FORMAT, basename, currPart++, totalParts )
                );
                overflowZipEntry = writeEntriesToPart( overflowZipEntry, zf, outFilePart, enumeration, approxPartSizeBytes );
                logger.debug( "Wrote {}", outFilePart );
            }
        }
    }

    /**
     * Write an entry to the to the outFilePart
     *
     * @param overflowZipEntry    ZipEntry that didn't fit in the last part, or null
     * @param inZipFile           The large archive to split
     * @param outFilePart         The part of the archive currently being worked on
     * @param enumeration         Enumeration of ZipEntries
     * @param approxPartSizeBytes Approximate part size
     * @return Overflow ZipEntry, or null
     * @throws IOException File access exceptions
     */
    private static ZipEntry writeEntriesToPart(
        @Nullable ZipEntry overflowZipEntry,
        @NotNull final ZipFile inZipFile,
        @NotNull final Path outFilePart,
        @NotNull final Enumeration<? extends ZipEntry> enumeration,
        final long approxPartSizeBytes
    ) throws IOException
    {
        try (
            ZipOutputStream zos =
                new ZipOutputStream( new FileOutputStream( outFilePart.toFile(), false ) )
        )
        {
            long partSize = 0;
            byte[] buffer = new byte[DEFAULT_BUFFER_SIZE];
            while ( overflowZipEntry != null || enumeration.hasMoreElements() )
            {
                ZipEntry entry = overflowZipEntry != null ? overflowZipEntry : enumeration.nextElement();
                overflowZipEntry = null;

                long entrySize = entry.getCompressedSize();
                if ( partSize + entrySize > approxPartSizeBytes )
                {
                    if ( partSize != 0 )
                    {
                        return entry;    // Finished this part, but return the dangling ZipEntry
                    }
                    // Add the entry anyway if the part would otherwise be empty
                }
                partSize += entrySize;
                zos.putNextEntry( entry );

                // Get the input stream for this entry and copy the entry
                try ( InputStream is = inZipFile.getInputStream( entry ) )
                {
                    int bytesRead;
                    while ( (bytesRead = is.read( buffer )) != -1 )
                    {
                        zos.write( buffer, 0, bytesRead );
                    }
                }
            }
            return null;    // Finished splitting
        }
    }

答案 4 :(得分:0)

这是我的解决方案:

public abstract class ZipHelper {

    public static NumberFormat formater = NumberFormat.getNumberInstance(new Locale("pt", "BR"));

    public static List<Path> zip(Collection<File> inputFiles, long maxSize) throws IOException {

        byte[] buffer = new byte[1024];
        int count = 0;
        long currentZipSize = maxSize;
        List<Path> response = new ArrayList<>();
        ZipOutputStream zip = null;
        for (File currentFile : inputFiles) {
            long nextFileSize = currentFile.length();
            long predictedZipSize = currentZipSize + nextFileSize;
            boolean needNewFile = predictedZipSize >= maxSize;
            System.out.println("[=] ZIP current (" + formater.format(currentZipSize) + ") + next file (" + formater.format(nextFileSize) + ") = predicted (" + formater.format(predictedZipSize) + ") > max (" + formater.format(maxSize) + ") ? " + needNewFile);
            if (needNewFile) {
                safeClose(zip);
                Path tmpFile = Files.createTempFile("teste-", (".part." + count++ + ".zip"));
                System.out.println("[#] Starting new file: " + tmpFile);
                zip = new ZipOutputStream(Files.newOutputStream(tmpFile));
                zip.setLevel(Deflater.BEST_COMPRESSION);
                response.add(tmpFile);
                currentZipSize = 0;
            }
            ZipEntry zipEntry = new ZipEntry(currentFile.getName());
            System.out.println("[<] Adding to ZIP: " + currentFile.getName());
            zip.putNextEntry(zipEntry);
            FileInputStream in = new FileInputStream(currentFile);
            zip.write(in.readAllBytes());
            zip.closeEntry();
            safeClose(in);
            long compressed = zipEntry.getCompressedSize();
            System.out.println("[=] Compressed current file: " + formater.format(compressed));
            currentZipSize += zipEntry.getCompressedSize();
        }
        safeClose(zip);
        return response;
    }

    public static void safeClose(Closeable... closeables) {
        if (closeables != null) {
            for (Closeable closeable : closeables) {
                if (closeable != null) {
                    try {
                        System.out.println("[X] Closing: (" + closeable.getClass() + ") - " + closeable);
                        closeable.close();
                    } catch (Throwable ex) {
                        System.err.println("[!] Error on close: " + closeable);
                        ex.printStackTrace();
                    }
                }
            }
        }
    }
}

和控制台输出:

[?] Files to process: [\data\teste\TestFile(1).pdf, \data\teste\TestFile(2).pdf, \data\teste\TestFile(3).pdf, \data\teste\TestFile(4).pdf, \data\teste\TestFile(5).pdf, \data\teste\TestFile(6).pdf, \data\teste\TestFile(7).pdf]
[=] ZIP current (3.145.728) + next file (1.014.332) = predicted (4.160.060) > max (3.145.728) ? true
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-3319961516431535912.part.0.zip
[<] Adding to ZIP: TestFile(1).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@3d99d22e
[=] Compressed current file: 940.422
[=] ZIP current (940.422) + next file (1.511.862) = predicted (2.452.284) > max (3.145.728) ? false
[<] Adding to ZIP: TestFile(2).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@49fc609f
[=] Compressed current file: 1.475.178
[=] ZIP current (2.415.600) + next file (2.439.287) = predicted (4.854.887) > max (3.145.728) ? true
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@cd2dae5
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-8849887746791381380.part.1.zip
[<] Adding to ZIP: TestFile(3).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@4973813a
[=] Compressed current file: 2.374.718
[=] ZIP current (2.374.718) + next file (2.385.447) = predicted (4.760.165) > max (3.145.728) ? true
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@6321e813
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-6305809161676875106.part.2.zip
[<] Adding to ZIP: TestFile(4).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@79be0360
[=] Compressed current file: 2.202.203
[=] ZIP current (2.202.203) + next file (292.918) = predicted (2.495.121) > max (3.145.728) ? false
[<] Adding to ZIP: TestFile(5).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@22a67b4
[=] Compressed current file: 230.491
[=] ZIP current (2.432.694) + next file (4.197.512) = predicted (6.630.206) > max (3.145.728) ? true
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@57855c9a
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-17160527941340008316.part.3.zip
[<] Adding to ZIP: TestFile(6).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@3b084709
[=] Compressed current file: 3.020.115
[=] ZIP current (3.020.115) + next file (1.556.237) = predicted (4.576.352) > max (3.145.728) ? true
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@3224f60b
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-14050058835776413808.part.4.zip
[<] Adding to ZIP: TestFile(7).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@63e2203c
[=] Compressed current file: 1.460.566
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@1efed156
[>] Generated ZIP files(s): [C:\Users\Cassio\AppData\Local\Temp\teste-3319961516431535912.part.0.zip, C:\Users\Cassio\AppData\Local\Temp\teste-8849887746791381380.part.1.zip, C:\Users\Cassio\AppData\Local\Temp\teste-6305809161676875106.part.2.zip, C:\Users\Cassio\AppData\Local\Temp\teste-17160527941340008316.part.3.zip, C:\Users\Cassio\AppData\Local\Temp\teste-14050058835776413808.part.4.zip]