Question

我正在尝试从7台计算机上载大量文件。在每台机器上，我运行6个线程以上传到S3中。当我从一台计算机上运行上载时，它工作正常，但是当我在7台计算机中运行时，它开始出现故障。

我在其余机器上都遇到错误。

错误-AmazonClientException com.amazonaws.SdkClientException：无法执行HTTP请求：等待池中的连接超时

我上传到S3的小文件总数= 1659328

每个线程中的记录数= 276554

那我必须关闭TransferManager吗？如果是，那我应该如何关闭它？我的应用程序是多线程的。当我调用tm.shutdownNow();时，其他线程将无法使用它。

这是我要上传到S3中的代码。

AWSCredentials credential = new ProfileCredentialsProvider("skjfffkjg-Prod-ServiceUser").getCredentials();
        AmazonS3Client s3Client = (AmazonS3Client) AmazonS3ClientBuilder.standard().withRegion("us-east-1")
                .withCredentials(new AWSStaticCredentialsProvider(credential)).withForceGlobalBucketAccessEnabled(true)
                .build();

        s3Client.getClientConfiguration().setMaxConnections(100);

上传到S3方法

public void uploadToToS3() {
        _logger.info("Number of record to be processed in current thread: : " + records.size());


        TransferManager tm = new TransferManager(s3Client);

        MultipleFileUpload upload = tm.uploadFileList(bucketName, "", new File(fileLocation), records);

        if (upload.isDone() == false) {
            System.out.println("Transfer: " + upload.getDescription());
            System.out.println("  - State: " + upload.getState());
            System.out.println("  - Progress: " + upload.getProgress().getBytesTransferred());
        }
        try {
            upload.waitForCompletion();
        } catch (AmazonServiceException e1) {
            _logger.error("AmazonServiceException " + e1.toString());
        } catch (AmazonClientException e1) {
            _logger.error("AmazonClientException " + e1.toString());
        } catch (InterruptedException e1) {
            _logger.error("InterruptedException " + e1.toString());
        }
        System.out.println("Is Upload completed Successfully ="+upload.isDone());

        for (File file : records) {
            try {
                Files.delete(FileSystems.getDefault().getPath(file.getAbsolutePath()));
            } catch (IOException e) {
                _logger.error("IOException in file delete: " + e.toString());
                System.exit(1);
                _logger.error("IOException: " + e.toString());
            }
        }

        _logger.info("Calling Transfer manager shutdown");
        // tm.shutdownNow();
    }

我是否必须关闭任何内容才能顺利上传？

Answer 1

拥有S3对象时，需要中止并关闭它-在中止打开的连接或关闭文件读取器等时

因此，您需要确保正确关闭对象请求。顺便说一句，增加最大连接数可能不是解决此问题的最佳方法。

有一个AWS API，允许中止S3操作。我相信您需要添加“ finally”块，以便在异常期间控制上传。

以下是我要使用的代码段：

public void uploadToToS3() {
        _logger.info("Number of record to be processed in current thread: : " + records.size());
    TransferManager tm = new TransferManager(s3Client);

    MultipleFileUpload upload = tm.uploadFileList(bucketName, "", new File(fileLocation), records);

    if (upload.isDone() == false) {
        System.out.println("Transfer: " + upload.getDescription());
        System.out.println("  - State: " + upload.getState());
        System.out.println("  - Progress: " + upload.getProgress().getBytesTransferred());
    }
    try {
        upload.waitForCompletion();
    } catch (AmazonServiceException e1) {
        _logger.error("AmazonServiceException " + e1.toString());
    } catch (AmazonClientException e1) {
        _logger.error("AmazonClientException " + e1.toString());
    } catch (InterruptedException e1) {
        _logger.error("InterruptedException " + e1.toString());
    } 

    // ************ THIS IS THE MODIFICATION ************ 
    finally {
        upload.getSubTransfers().forEach(s -> s.abort());
    }
    // *************************************************** 

    System.out.println("Is Upload completed Successfully ="+upload.isDone());
    for (File file : records) {
        try {
            Files.delete(FileSystems.getDefault().getPath(file.getAbsolutePath()));
        } catch (IOException e) {
            _logger.error("IOException in file delete: " + e.toString());
            System.exit(1);
            _logger.error("IOException: " + e.toString());
        }
    }

    _logger.info("Calling Transfer manager shutdown");
    // tm.shutdownNow();
}

超时等待来自池的连接以上传S3

1 个答案: