Question

我使用以下代码从Amazon S3读取文本文件，并逐行处理。这段代码有效，但问题是它很慢。

 GetObjectRequest getObjRequest = new GetObjectRequest()
    .WithBucketName(amazonSettings.BucketName)
    .WithKey(_fileKey);
using (AmazonS3 client = AWSClientFactory.CreateAmazonS3Client(
    amazonSettings.AccessKey, 
    amazonSettings.SecretAccessKey))
using (GetObjectResponse getObjRespone = client.GetObject(getObjRequest))
using (Stream amazonStream = getObjRespone.ResponseStream)
{                        
    StreamReader amazonStreamReader = new StreamReader(amazonStream);
    tempGsContact = new GSContact();
    while ((_fileLine = amazonStreamReader.ReadLine()) != null)
    {
        if (_fileLine.Equals("END:VCARD"))
        {
            // Make process 1
        }
        else if (!_fileLine.Equals(string.Empty))
        {
            //Make process 2
        }
    }                        
}

问题是：我能否获得更充分的方法来降低时间成本？

Answer 1

.NET中的HTTPWebResponse存在类似的性能瓶颈，这可能是他们制作的AmazonS3类所包含的。

这是由于对象需要很长时间才能解析代理设置，列出了一些可能的解决方案here，但最简单的选择可能是将以下内容添加到app.config文件中：

<system.net>
  <defaultProxy enabled="false">
    <proxy/>
    <bypasslist/>
    <module/>
  </defaultProxy>
</system.net>

或者你可以在这里替换这个电话：

AWSClientFactory.CreateAmazonS3Client(amazonSettings.AccessKey, amazonSettings.SecretAccessKey)

调用overload接受'AmazonS3Config'的第三个参数，您可以通过'AmazonS3Config.ProxyHost = null'指定空代理 - 这应该与上面的配置更改有效仅用于该请求。

Answer 2

此功能用于计算文本文件中的字数。

private int WordCount(string awsBucketName, string awsFilePath, string wordForSearch)
    {
        string line =string.Empty;
        int counter = 0;
        if ((cloudKaseClient != null) & (token == tokenCfg))
        {
            var request = new GetObjectRequest()
            {
                BucketName = awsBucketName,
                Key = awsFilePath
            };
             using (var response = cloudKaseClient.GetObject(request))
            {
                    StreamReader reader = new StreamReader(response.ResponseStream);
                    while ((line = reader.ReadLine()) != null)
                    {
                        counter += line.Split(' ').Where(t => t.ToString().ToLower().Contains(wordForSearch.ToLower())).Count();
                    }
                    reader.Close();             
            }
        }
        return counter;
    }

Answer 3

或者你也可以使用这个功能，

    private static void ReadS3Object(AmazonS3Client client)
    {
        GetObjectRequest request = new GetObjectRequest();

        request.WithBucketName(BUCKET_NAME);
        request.WithKey(S3_KEY);

        GetObjectResponse response = client.GetObject(request);

        StreamReader reader = new StreamReader(response.ResponseStream);

        String content = reader.ReadToEnd();

        Console.Out.WriteLine("Read S3 object with key " + S3_KEY + " in bucket " + BUCKET_NAME + ". Content is: " + content);
    }

从亚马逊s3流中读取文本

3 个答案: