我想从 Azure blob 存储下载所有文件,从这些文件中创建一个 zip 文件,然后将 zip 文件上传回 blob 存储。 由于文件大小可能非常大,我不想最大化内存。 此外,此操作需要非常快。
JAVA SDK - azure-storage-blob 12.8.0
编辑:到目前为止编写的代码。不确定如何进一步并行上传管道输入流数据。
String zipFileName = formFileName(exportRequest, requestId);
final PipedOutputStream pipedOutputStream = new PipedOutputStream();
final PipedInputStream pipedInputStream = new PipedInputStream(pipedOutputStream);
AzureObjectStoreService objectStoreService =managedObjectStoreUtils.getObjectStoreService();
if (filesToZip.size() > 0) {
System.out.println("Files to zip "+ filesToZip.size());
CompletableFuture<Boolean> zipCreationFuture = CompletableFuture.runAsync(() -> {
LoggerHelper.logInfo(logger, "Inside createZIP file async function");
ZipOutputStream zipOutputStream = new ZipOutputStream(pipedOutputStream);
try {
for (String fileName : filesToZip) {
try {
BlobClient blobClient = objectStoreService.getBlobContainerClient().getBlobClient(fileName);
LoggerHelper.logInfo(logger, "Adding zipEntry for file : " + fileName);
final ZipEntry zipEntry = new ZipEntry(fileName);
zipOutputStream.putNextEntry(zipEntry);
byte[] buffer;
ByteArrayOutputStream output = new ByteArrayOutputStream();
buffer= output.toByteArray();
blobClient.getBlockBlobClient().download(output);
int len;
while ((len = buffer.length) > 0) {
zipOutputStream.write(buffer, 0, len);
}
zipOutputStream.closeEntry();
} catch (SdkClientException e) {
LoggerHelper.logExceptionWithMessage(logger, this.getClass().getName(), (Exception) e);
LoggerHelper.logError(logger, "Failed while getting s3 object");
}
}
zipOutputStream.finish();
} catch (IOException ex) {
LoggerHelper.logExceptionWithMessage(logger, this.getClass().getName(), (Exception) ex);
LoggerHelper.logError(logger, "Creating zip file failed");
} finally {
try {
zipOutputStream.close();
} catch (IOException e) {
LoggerHelper.logExceptionWithMessage(logger, this.getClass().getName(), (Exception) e);
LoggerHelper.logError(logger, "Failed to close the zip output stream");
}
}
LoggerHelper.logInfo(logger, "Completed createZIP file async function");
// return true;
}).handle((o, exception) -> {
LoggerHelper.logExceptionWithMessage(logger, this.getClass().getName(), (Exception) exception);
LoggerHelper.logError(logger, "Creating zip file failed");
return null;
});
答案 0 :(得分:1)
能够以这种方式做到这一点。如果有人有更好的方法,请告诉我。
CompletableFuture.runAsync(() -> {
BlobClient blobClient = objectStoreService.getBlobContainerClient().getBlobClient(zipFileName);
BlobOutputStream blobOutputStream = blobClient.getBlockBlobClient().getBlobOutputStream();
try {
int nextData= pipedInputStream.read();
while (nextData!=-1) {
blobOutputStream.write(nextData);
nextData = pipedInputStream.read();
}blobOutputStream.close();
} catch (IOException e) {
e.printStackTrace();
}