Question

因此，我很难找到将文件下载流式传输到s3存储桶的正确方法，而无需将其写入磁盘或将其全部保存在内存中。我有一个解决方案，我从HttpUrlConnection对象生成InputStream，然后将其传递给s3 putObject调用，如下所示：

HttpURLConnection httpConn = (HttpURLConnection) resourceUrl.openConnection();
httpConn.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
httpConn.connect();
int contentLength = httpConn.getContentLength();
try (InputStream is = httpConn.getInputStream()) {
    ObjectMetadata objectMetadata = new ObjectMetadata();
    objectMetadata.setContentLength(contentLength);
    PutObjectRequest req = new PutObjectRequest(bucketname,url.substring(url.lastIndexOf("/")),is,objectMetadata);
    s3client.putObject(req);
}

除了它是阻塞之外，这是有效的，所以当你提交带有资源的url来拉下它时，它会远离服务器端，但需要很长时间才能完成（文件是500 mb +），POST请求以500甚至结束虽然它运作成功。

所以我厌倦了移动到CompletableFuture对象尝试让它在自己的线程中运行。我一直试图从我在网上找到的不同来源拼凑代码，然后来到这里：

try (InputStream is = httpConn.getInputStream()) {
ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setContentLength(contentLength);
PutObjectRequest req = new PutObjectRequest(bucketname,url.substring(url.lastIndexOf("/")),is,objectMetadata);
CompletableFuture<PutObjectResult> future = CompletableFuture.supplyAsync(() ->
    s3client.putObject(req), Executors.newScheduledThreadPool(1));
future.whenComplete((resp, err) -> {
    try {
    if (resp != null) {
        System.out.println(resp);
    } else {
        // Handle error
        err.printStackTrace();
    }
    } finally {
        // Lets the application shut down. Only close the client when you are completely done with it.
        s3client.shutdown();
    }
});
...

上面的代码会导致以下堆栈跟踪：

java.util.concurrent.CompletionException：com.amazonaws.SdkClientException：无法执行HTTP请求：流已关闭           at java.util.concurrent.CompletableFuture.encodeThrowable（CompletableFuture.java:273）           ...   引起：com.amazonaws.SdkClientException：无法执行HTTP请求：流已关闭           at com.amazonaws.http.AmazonHttpClient $ RequestExecutor.handleRetryableException（AmazonHttpClient.java:1114）           在   的 appback.MyResource.lambda $ $ processNewMachine 0（MyResource.java:93）           ......还有7个   引起：java.io.IOException：流已关闭           在sun.net.www.protocol.http.HttpURLConnection $ HttpInputStream.ensureOpen（HttpURLConnection.java:3348)
  ......还有18个

第93行是对 s3client.putObject（req）的调用

我已经开始根据我在谷歌搜索时发现的一个开源项目编写代码片段

    @Override
public CompletableFuture<PutObjectResult> putObject(String bucketName, String key, InputStream inputStream,
        int length) {
    ObjectMetadata objectMetadata = new ObjectMetadata();
    objectMetadata.setContentLength(length);
    ...
    PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName, key, inputStream, objectMetadata);
    return CompletableFuture.supplyAsync(() -> s3Client.putObject(putObjectRequest), executorService);
}

他们几乎完全相同的事实让我觉得我对如何将InputStreams传递给CompletableFuture调用有一个错误的假设。如果您对如何解决此问题有任何见解，请与我们联系。谢谢！

尝试从http文件下载InputStream读取时，CompleteableFuture抛出IOException

0 个答案: