Question

大约两年来，我一直成功地使用一段代码从网站下载页面，以便我可以解析内容并将其产品价格加载到我自己的网站中。突然，这已停止工作，取而代之的是，我得到此“必须启用javascript”的信息，其余页面丢失了。

1）他们的服务器是否有可能识别我的搜寻器并将其重定向？

2）他们是否只是简单地修改了自己的网站，以便现在通过javascript例程呈现HTML？

3）有没有一种方法可以强制下载过程仅下载渲染的javascript。即首先强制运行javascript并下载生成的HTML？

我正在抓取的网站为http://www.thebeerstore.ca/beers/search

这是下载的结果文本。

<html><head>
<title>You are being redirected...</title>
<noscript>Javascript is required. Please enable javascript before you   are allowed to see this page.</noscript>
<script>
    (bunch of unreadable coded script here )
</script>

下面是我的代码。谢谢您的帮助！

 fileprivate func execute()
{
    if !executed
    {
        executed = true

        let endpoint = URL(string: url!)

        if (endpoint == nil) { return }

        let request = NSMutableURLRequest(url: endpoint!)

        if data != nil
        {
            if data is NSData
            {
                request.httpMethod = "POST"
                request.httpBody = data as? Data
            }
            else
            {
                request.httpMethod = "POST"
                request.httpBody = try! JSONSerialization.data(withJSONObject: data!, options: [])
                request.addValue("application/json", forHTTPHeaderField: "Content-Type")
                request.addValue("application/json", forHTTPHeaderField: "Accept")
            }
        }
        request.timeoutInterval = 30

        let session = URLSession(configuration: URLSessionConfiguration.ephemeral)

        let task = session.dataTask(with: request as URLRequest, completionHandler: {
            data, response, error in

            if error == nil
            {
                WebRequestManager.pop()
                self.completion?(data,response,error)
            }
            else
            {
                if self.trials < WebRequest.MaxTrials
                {
                    self.trials += 1
                    self.executed = false
                    self.execute()
                }
                else
                {
                    WebRequestManager.pop()
                    self.completion?(data,response,error)
                }
            }
        })            

        task.resume()
    }

}

下载网页内容

0 个答案: