Question

我正在尝试使用asp.net中的Multipart上传方法上传视频文件Amazon S3，并使用日志跟踪上传进度。它每次上传106496，一次只运行一个线程。我没注意到多个线程在运行。请澄清一下为什么它运行单线程并且即使是20Mb文件也需要很长时间才能上传它需要大约2分钟。

这是我的代码，使用UploadPartRequest。

private void UploadFileOnAmazon(string subUrl, string filename, Stream audioStream, string extension)
    {
        client = new AmazonS3Client(accessKey, secretKey, Amazon.RegionEndpoint.USEast1);

        // List to store upload part responses.
        List<UploadPartResponse> uploadResponses = new List<UploadPartResponse>();


        // 1. Initialize.
        InitiateMultipartUploadRequest initiateRequest = new InitiateMultipartUploadRequest
        {
            BucketName = bucketName,
            Key = subUrl + filename
        };

        InitiateMultipartUploadResponse initResponse =
            client.InitiateMultipartUpload(initiateRequest);

        // 2. Upload Parts.
        //long contentLength = new FileInfo(filePath).Length;
        long contentLength = audioStream.Length;
        long partSize = 5 * (long)Math.Pow(2, 20); // 5 MB


        try
        {
            long filePosition = 0;
            for (int i = 1; filePosition < contentLength; i++)
            {

                UploadPartRequest uploadRequest = new UploadPartRequest
                {
                    BucketName = bucketName,
                    Key = subUrl + filename,
                    UploadId = initResponse.UploadId,
                    PartNumber = i,
                    PartSize = partSize,
                    FilePosition = filePosition,
                    InputStream = audioStream
                    //FilePath = filePath
                };

                // Upload part and add response to our list.
                uploadRequest.StreamTransferProgress += new EventHandler<StreamTransferProgressArgs>(UploadPartProgressEventCallback);                 
                uploadResponses.Add(client.UploadPart(uploadRequest));

                filePosition += partSize;
            }

            logger.Info("Done");


            // Step 3: complete.
            CompleteMultipartUploadRequest completeRequest = new CompleteMultipartUploadRequest
            {
                BucketName = bucketName,
                Key = subUrl + filename,
                UploadId = initResponse.UploadId,
                //PartETags = new List<PartETag>(uploadResponses)

            };
            completeRequest.AddPartETags(uploadResponses);
            CompleteMultipartUploadResponse completeUploadResponse =
                client.CompleteMultipartUpload(completeRequest);

        }
        catch (Exception exception)
        {
            Console.WriteLine("Exception occurred: {0}", exception.Message);
            AbortMultipartUploadRequest abortMPURequest = new AbortMultipartUploadRequest
            {
                BucketName = bucketName,
                Key = subUrl + filename,
                UploadId = initResponse.UploadId
            };
            client.AbortMultipartUpload(abortMPURequest);
        }
    }
    public static void UploadPartProgressEventCallback(object sender, StreamTransferProgressArgs e)
    {
        // Process event. 
        logger.DebugFormat("{0}/{1}", e.TransferredBytes, e.TotalBytes);
    }

我的代码有什么问题，或者如何让线程同时运行以加快上传速度？

Answer 1

不要自己管理分段上传，而是尝试使用为您完成所有艰苦工作的TransferUtility！

请参阅：Using the High-Level .NET API for Multipart Upload

Answer 2

AmazonS3Client在内部使用AmazonS3Config实例来了解用于传输的缓冲区大小（ref 1）。此AmazonS3Config（ref 2）有一个名为BufferSize的属性，其默认值是从AWSSDKUtils（ref 3）中的常量中检索的 - 在当前的SDK版本中默认为8192字节 - 相当小的值恕我直言。

您可以使用AmazonS3Config的自定义实例，其值为BufferSize。要构建一个尊重自定义配置的AmazonS3Client实例，必须将自定义配置传递给客户端构造函数。例如：

// Create credentials.
AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
// Create custom config.
AmazonS3Config config = new AmazonS3Config
{
    RegionEndpoint = Amazon.RegionEndpoint.USEast1,
    BufferSize = 512 * 1024, // 512 KiB
};
// Pass credentials + custom config to the client.
AmazonS3Client client = new AmazonS3Client(credentials, config);

// They uploaded happily ever after.

Amazon S3 Multipart UploadPartRequest仅允许使用asp.net同时上传单个线程

2 个答案: