Question

我正在尝试从BOM Australia获取天气数据。手动方式是转到http://www.bom.gov.au/jsp/ncc/cdio/weatherData/av?p_nccObsCode=136&p_display_type=dailyDataFile&p_startYear=&p_c=&p_stn_num=2064并点击“所有年份的数据”，然后下载文件！

以下是我尝试自动化的内容：

using (WebClient client = new WebClient())
            {

                string html = client.DownloadString("http://www.bom.gov.au/jsp/ncc/cdio/weatherData/av?p_nccObsCode=136&p_display_type=dailyDataFile&p_startYear=&p_c=&p_stn_num=2064");


                List<string> list = LinkExtractor.Extract(html);
                foreach (var link in list)
                {
                    if (link.StartsWith("/jsp/ncc/cdio/weatherData/av?p_display_type=dailyZippedDataFile"))
                    {

                        string resource = "http://www.bom.gov.au" + link;
                        MessageBox.Show(resource);


                        client.DownloadFileAsync(new Uri(resource), Dts.Connections["data.zip"].ConnectionString);
                        break;
                    }
                }




            }

不要担心linkExtractor，它可以正常工作，因为我能够看到提供该文件的链接。问题是'DownloadFileAsync'创建了一个新请求，由于文件需要相同的会话，因此不允许下载文件。

有没有办法可以做到这一点？请联系以获得更多说明。

更新：

以下是我使用来自HttpWebRequest的Cookie所做的更改。但是，我仍然无法下载该文件。

HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.bom.gov.au/jsp/ncc/cdio/weatherData/av?p_nccObsCode=136&p_display_type=dailyDataFile&p_startYear=&p_c=&p_stn_num=2064");
            request.CookieContainer = new CookieContainer();

            HttpWebResponse response = (HttpWebResponse)request.GetResponse();

            foreach (Cookie cook in response.Cookies)
            {
                MessageBox.Show(cook.ToString());
            }

            if (response.StatusCode == HttpStatusCode.OK)
           {
                Stream receiveStream = response.GetResponseStream();
                StreamReader readStream = null;

                if (response.CharacterSet == null)
                {
                    readStream = new StreamReader(receiveStream);
                }
                else
                {
                    readStream = new StreamReader(receiveStream, Encoding.GetEncoding(response.CharacterSet));
                }

                string data = readStream.ReadToEnd();



                using (WebClient client = new WebClient())
                {
                    foreach (Cookie cook in response.Cookies)
                    {
                        MessageBox.Show(cook.ToString());
                        client.Headers.Add(HttpRequestHeader.Cookie, cook.ToString());
                    }

                    List<string> list = LinkExtractor.Extract(data);
                    foreach (var link in list)
                    {
                        if (link.StartsWith("/jsp/ncc/cdio/weatherData/av?p_display_type=dailyZippedDataFile"))
                        {

                            string initial = "http://www.bom.gov.au" + link;
                            MessageBox.Show(initial);

                            //client.Headers.Add(HttpRequestHeader.Cookie, "JSESSIONID=2EBAFF7EFE2EEFE8140118CE5170B8F6");
                            client.DownloadFile(new Uri(initial), Dts.Connections["data.zip"].ConnectionString);
                            break;
                        }
                    }




                }

                response.Close();
                readStream.Close();
            }

Answer 1

你得到的html和其中的url是HtmlEncoded。这使得当你从html中对url进行子串，你需要解码它，理想情况下。这就是zip的下载URL：

   /jsp/ncc/cdio/weatherData/av?p_display_type=dailyZippedDataFile&amp;p_stn_num=2064&amp;p_c=-938623&amp;p_nccObsCode=136&amp;p_startYear=2016

有助手类为我们解码：WebUtility

此代码确实下载了zip文件：

using (var client = new WebClient())
{
    var url = "http://www.bom.gov.au/jsp/ncc/cdio/weatherData/av?p_nccObsCode=136&p_display_type=dailyDataFile&p_startYear=&p_c=&p_stn_num=2064";    
    string html = client.DownloadString(url);

    var pos = html.IndexOf("/jsp/ncc/cdio/weatherData/av?p_display_type=dailyZippedDataFile");
    var endpos = html.IndexOf('"', pos);
    string link = html.Substring(pos, endpos - pos);

    var decodedLink = WebUtility.HtmlDecode(link);
    string resource = "http://www.bom.gov.au" + decodedLink;                    


    client.DownloadFile(new Uri(resource), @"c:\temp\bom2.zip");

}

在这种情况下，您不需要保留Cookie，但您需要小心处理您解析的网址。

从动态生成的链接中下载文件，该链接位于HTML

1 个答案: