HttpWebResponse下载大文件时提前切断

时间:2014-05-14 23:13:17

标签: json httpwebrequest httpwebresponse usps epf

我正在使用USPS EPF(电子产品履行)提供的C#Client类,以便通过控制台应用程序下载USPS文件。我使用我的控制台应用程序登录我的USPS凭据,指定我要下载的文件,并获取该文件。所有这些都适用于我们可以访问的两个较小的文件(AMS Developer Kit,47.7 MB和DPV Developer Kit,1.59 MB)。但是,当我尝试下载2.8 GB AMS商业DVD文件时,这是我真正关心的唯一一个,我遇到了问题。控制台应用程序每次停止以1.75 GB的速度下载文件。由于它是一个.tar文件,我可以打开它并看到一些内容,但自然会丢失很多。 USPS提供的Client类不会抛出任何异常或错误;它应该读到文件的末尾,但它只是提前停止。

我已经尝试了我能想到的一切:改变HttpWebRequest属性(将KeepAlive更改为true,增加Timeout值),修改getEpfFile方法以使用IsolatedStorageFile而不是MemoryStream来获取文件,甚至检查我们的网络人们要确保没有任何网络设置导致超时。我尝试从我的机​​器和不同的网络服务器下载相同的结果。我考虑使用WebClient,但这需要下载文件的整个URL的参数,这是未知的。据我所知,我必须使用HttpWebRequest来访问USPS EPF文件。

这是USPS提供的Client.cs类的getEpfFile方法(我为任何格式问题道歉,这是我在网站上的第一篇文章):

//获取实际文件

    public bool getEpfFile(String fileid)
    {
        bool downloadSuccess = true;
        string strUrl = this.strBaseUrl + "/download/epf";

        try
        {

            Console.WriteLine("Starting file download ...");

            // add json to URL
            Dictionary<string, string> json_value = new Dictionary<string, string>();
            json_value.Add("logonkey", this.logon_key);
            json_value.Add("tokenkey", this.token_key);
            json_value.Add("fileid", fileid);

            JavaScriptSerializer jsonSerializer = new JavaScriptSerializer();
            string json_string = "obj=" + jsonSerializer.Serialize(json_value);

            System.Text.ASCIIEncoding encoding = new System.Text.ASCIIEncoding();
            Byte[] byteArray = encoding.GetBytes(json_string);

            // set URL              
            Uri address = new Uri(strUrl);

            // web request  
            HttpWebRequest request = WebRequest.Create(address) as HttpWebRequest;

            request.UserAgent = "USPS .NET Sample";
            request.KeepAlive = false;
            request.Timeout = 100000;

            request.ProtocolVersion = HttpVersion.Version10;

            request.Method = "POST";
            request.ContentLength = byteArray.Length;
            request.ContentType = "application/x-www-form-urlencoded";

            // add headers
            // request.Headers.Add("Akamai-File-Request", filepath);
            request.Headers.Add("logonkey", this.logon_key);
            request.Headers.Add("tokenkey", this.token_key);
            request.Headers.Add("fileid", fileid);

            // post request
            Stream dataStream = request.GetRequestStream();
            dataStream.Write(byteArray, 0, byteArray.Length);
            dataStream.Close();

            // Get response  
            HttpWebResponse response = request.GetResponse() as HttpWebResponse;


            if (request.HaveResponse == true && response != null)
            {

                Stream remoteStream = response.GetResponseStream();

                Directory.CreateDirectory("c:\\Atemp");
                Stream localStream = File.Create("c:\\Atemp\\dd.tar");

                //byte[] buffer = new byte[2048];
                byte[] buffer = new byte[1000000];

                int bytesRead = 0;

                do
                {
                    // Read data (up to 1k) from the stream
                    bytesRead = remoteStream.Read(buffer, 0, buffer.Length);

                    // Write the data to the local file
                    localStream.Write(buffer, 0, bytesRead);

                } while (bytesRead > 0);


                int i = response.Headers.Count;


                for (int x = 0; x < i; x++)
                {

                    if (response.Headers.Keys[x].ToString() == "User-Tokenkey")
                    {
                        this.token_key = response.Headers[x].ToString();
                    }
                    else if (response.Headers.Keys[x].ToString() == "User-Logonkey")
                    {
                        this.logon_key = response.Headers[x].ToString();
                    }

                    else if (response.Headers.Keys[x].ToString() == "Service-Response")
                    {
                        Console.WriteLine("Web service result: " + response.Headers[x].ToString());
                    }
                    else if (response.Headers.Keys[x].ToString() == "Service-Messages")
                    {
                        Console.WriteLine("Resulting Messages: " + response.Headers[x].ToString());
                    }

                }

                // close resources
                localStream.Close();
                remoteStream.Close();
                response.Close();

                Console.WriteLine("File Download completed.");
            }
        }
        catch (Exception ex)
        {
            downloadSuccess = false;
            string str = ex.Message;
            str += "";
        }

        return downloadSuccess;
    }

任何关于它为什么一直提前切断的见解都将受到大力赞赏。

3 个答案:

答案 0 :(得分:1)

原来,USPS一直有一个下载管理器程序,这使我想要做的事情变得容易了10倍。我通过USPS代表直接通过电子邮件发送了zip文件,但如果有人需要,可以在此处获取文档(包括下载链接):https://ribbs.usps.gov/acs/documents/tech_guides/ACS_EPF_DownloadManagerTechnicalGuide.pdf

答案 1 :(得分:0)

您的超时时间只设置为超过一分钟......您能快速下载吗?增加超时时间再试一次。 (提防它以毫秒为单位)

答案 2 :(得分:0)

前段时间我遇到了同样的问题,原来该文件实际上是在AKAMAI服务器上而不是EPF服务器上导致问题的原因。它更多的是2 GB,那么它最有可能在AKAMAI服务器上,所以我给出的代码对我有用,如下所示,

 public static bool DownloadEPFByFileIDUsingNewList(ref USPSUserDTO obj, USPSProductsDTO productList)
    {
        USPSFileDTO userDTO = new USPSFileDTO();
        if (obj != null)
        {
            foreach (var product in productList.fileList)
            {
                product.productcode = "NCAW";
                product.productid = "NCL18H";
                downloadSelectedFile(product, ref obj);
            }
        }
        return true;
    }


     private static void downloadSelectedFile(USPSProductInfo uspsProductDownload, ref USPSUserDTO obj)
    {
        string serviceResponse = string.Empty;
        string serviceMessages = string.Empty;
        USPSProductInfo uspsProductInfo = uspsProductDownload;
        int downloadPercentage = 0;
        bool isAkamaiFile = IsAkamaiFile(uspsProductDownload.productcode); //true;

        try
        {
            string[] statusResponse = SetStatus(USPS_URL, VERSION, obj.logonkey, obj.tokenkey, "S", uspsProductInfo.fileid, obj.login, obj.pword);
            if (statusResponse != null)
            {
                obj.logonkey = statusResponse[0];
                obj.tokenkey = statusResponse[1];
            }
            Uri StatusURL = new Uri(USPS_URL + (isAkamaiFile ? "/download/file" : "/download/epf"));
            byte[] byteLength = new ASCIIEncoding().GetBytes("obj=" + new JavaScriptSerializer().Serialize((object)new Dictionary<string, string>()
            {
                {
                  "logonkey",
                  obj.logonkey
                },
                {
                  "tokenkey",
                  obj.tokenkey
                },
                {
                  "fileid",
                  uspsProductInfo.fileid
                }
            }));

            HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create(StatusURL);
            httpWebRequest.UserAgent = "EPF Download Manager - " + VERSION;
            httpWebRequest.KeepAlive = false;
            httpWebRequest.Timeout = 100000;
            httpWebRequest.Method = METHOD_TYPE;
            httpWebRequest.ContentLength = (long)byteLength.Length;
            httpWebRequest.ContentType = CONTENT_TYPE;
            (httpWebRequest.Headers).Add("Akamai-File-Request", uspsProductInfo.filepath + uspsProductInfo.filename);
            (httpWebRequest.Headers).Add("logonkey", obj.logonkey);
            (httpWebRequest.Headers).Add("tokenkey", obj.tokenkey);
            (httpWebRequest.Headers).Add("fileid", uspsProductInfo.fileid);

            //TODO chage path 
            string path = @"C:\try\newZipFiles" + "\\" + uspsProductInfo.fulfilled;
            if (!Directory.Exists(path))
                Directory.CreateDirectory(path);

            string outputFilePath = path + "\\" + uspsProductInfo.filename;

            Stream stream = ((WebRequest)httpWebRequest).GetRequestStream();
            stream.Write(byteLength, 0, byteLength.Length);
            stream.Close();
            HttpWebResponse httpWebResponse = httpWebRequest.GetResponse() as HttpWebResponse;
            int headercount = httpWebResponse.Headers.Count;
            for (int headCount = 0; headCount < headercount; ++headCount)
            {
                if ((httpWebResponse.Headers.Keys[headCount]).ToString() == "User-Tokenkey")
                {
                    string responseToken = (httpWebResponse.Headers)[headCount].ToString();
                    if (responseToken.Contains(","))
                    {
                        responseToken = responseToken.Replace(",", "").Trim();
                    }
                    obj.tokenkey = responseToken.Trim();
                }
                else if ((httpWebResponse.Headers.Keys[headCount]).ToString() == "User-Logonkey")
                {
                    string responseLogonkey = ((httpWebResponse.Headers)[headCount]).ToString();
                    if (responseLogonkey.Contains(","))
                        responseLogonkey = responseLogonkey.Replace(",", "");
                    obj.logonkey = responseLogonkey.Trim();
                }
                else if ((httpWebResponse.Headers.Keys[headCount]).ToString() == "Service-Response")
                    serviceResponse = ((httpWebResponse.Headers)[headCount]).ToString().Replace(",", "").Trim();
                else if ((httpWebResponse.Headers.Keys[headCount]).ToString() == "Service-Messages")
                    serviceMessages = ((httpWebResponse.Headers)[headCount]).ToString();
            }

            if (serviceResponse == "success")
            {
                long countr = 0;
                Stream streamResponse = httpWebResponse.GetResponseStream();
                Stream fileStream = (Stream)System.IO.File.Create(outputFilePath);
                long downloadedFileSize = 0L;
                long originalFileSize = long.Parse(uspsProductInfo.filesize);
                byte[] bufferSize = isAkamaiFile ? new byte[1024] : new byte[512];
                int bytesRead = streamResponse.Read(bufferSize, 0, bufferSize.Length);
                try
                {
                    while (bytesRead > 0)
                    {
                        countr++;
                        fileStream.Write(bufferSize, 0, bytesRead);
                        downloadedFileSize += (long)bytesRead;
                        downloadPercentage = (int)(downloadedFileSize * 100L / originalFileSize);
                        bytesRead = streamResponse.Read(bufferSize, 0, bufferSize.Length);
                        Console.Clear();
                        Console.WriteLine("Downloaded " + downloadedFileSize + " of " + originalFileSize + " bytes." + " " + downloadPercentage + "%" + "counter" + countr);
                    }
                }
                catch (Exception)
                {
                }

                fileStream.Close();
                streamResponse.Close();
                httpWebResponse.Close();

                if (downloadedFileSize == long.Parse(uspsProductInfo.filesize))
                {
                    string[] statusCompleted = SetStatus(USPS_URL, VERSION, obj.logonkey, obj.tokenkey, "C", uspsProductInfo.fileid, obj.login, obj.pword);
                    if (statusCompleted != null)
                    {
                        obj.logonkey = statusCompleted[0];
                        obj.tokenkey = statusCompleted[1];
                    }
                }

            }

            ExtractRarFile(outputFilePath, uspsProductInfo.fulfilled, path);


        }
        catch (Exception exception_0)
        {

        }
        finally
        {

        }

    }


   public static bool IsAkamaiFile(string productCodeIn)
    {
        return productCodeIn == "AISVR" || productCodeIn == "AMS" || (productCodeIn == "NCAW" || productCodeIn == "NCAWM") || productCodeIn == "NCAM";
    }

检查您的产品代码并试一试,它可以根据您的产品代码进行一些调整。最好的运气。