Question

我想将一个目录从EMR本地文件系统上传到s3作为压缩文件。

有没有比我目前使用的方法更好的方法来解决这个问题？

是否可以将ZipOutputStream作为Reducer输出返回？

由于

zipFolderAndUpload("target", "target.zip", "s3n://bucketpath/");


static public void zipFolderAndUpload(String srcFolder, String zipFile, String dst) throws Exception {

    //Zips a directory
    FileOutputStream fileWriter = new FileOutputStream(zipFile);
    ZipOutputStream zip = new ZipOutputStream(fileWriter);
    addFolderToZip("", srcFolder, zip);
    zip.flush();
    zip.close();

    // Copies the zipped file to the s3 filesystem,
    InputStream in = new BufferedInputStream(new FileInputStream(zipFile));
    Configuration conf = new Configuration();
    FileSystem fs = FileSystem.get(URI.create(dst+zip), conf);
    OutputStream out = fs.create(new Path(dst+zip));
    IOUtils.copyBytes(in, out, 4096, true);

}

static private void addFileToZip(String path, String srcFile, ZipOutputStream zip) throws Exception {

    File folder = new File(srcFile);
    if (folder.isDirectory()) {
        addFolderToZip(path, srcFile, zip);
    } else {
        byte[] buf = new byte[1024];
        int len;
        FileInputStream in = new FileInputStream(srcFile);
        zip.putNextEntry(new ZipEntry(path + "/" + folder.getName()));
        while ((len = in.read(buf)) > 0) {
            zip.write(buf, 0, len);
        }
    }
}

static private void addFolderToZip(String path, String srcFolder, ZipOutputStream zip) throws Exception {
    File folder = new File(srcFolder);

    for (String fileName : folder.list()) {
        if (path.equals("")) {
            addFileToZip(folder.getName(), srcFolder + "/" + fileName, zip);
        } else {
            addFileToZip(path + "/" + folder.getName(), srcFolder + "/" + fileName, zip);
        }
    }
}

Answer 1

你采取的方法看起来很好。如果您发现它太慢，因为它是单线程的，那么您可以创建自己的写入zip文件的Hadoop OutputFormat实现。

您必须要注意的一点是，Java SE的ZipOutputFormat实现不支持Zip64，这意味着它不支持大于4GB的ZIP文件。还有其他ZIP实现，如TrueZIP。

将目录作为压缩文件从Elastic MapReduce上传到S3

1 个答案: