在Python中删除加载AJAX的网站

时间:2017-11-25 12:07:56

标签: javascript python json ajax api

我使用Selenium搜索了以下网站:https://www.eex-transparency.com/homepage/power/czech-republic/production/availability/non-usability/non-usability。我正在抓取所有表格数据。它运行良好,但运行脚本需要相当长的时间。因此,我开始寻找替代方案,并在StackOverflow上使用API​​向服务器发送请求,但是经过几个小时的尝试和搜索后,我放弃了,因为我没有得到好几件事:

  • 如何对API进行反向工程以发送正确的请求?
  • 我应该使用哪个网址?

这就是我提出的:

import json
import requests

url = "https://www.eex-transparency.com/ajax/en/navigation/ajaxGetNavi/12"

data = {
    "id": "16",
    "title": "Czech Republic",
    "url": "https:\\/\\/www.eex-transparency.com\\/homepage\\/power\\/czech-republic",
    "class": "country",
    "description": "",
    "children": [
      {
        "id": "649",
        "title": "Production",
        "url": False,
        "class": "",
        "description": "",
        "children": [
          {
            "id": "650",
            "title": "Capacity",
            "url": False,
            "class": "",
            "description": "",
            "children": [
              {
                "id": "651",
                "title": "Installed Capacity",
                "url": "https:\\/\\/www.eex-transparency.com\\/homepage\\/power\\/czech-republic\\/production\\/capacity\\/installed-capacity",
                "class": "",
                "description": ""
              }
            ]
          }
        ]
       }
      ]      
     }


response = requests.get(url, data=data)
file = response.json()

一般情况下,也许有人可以解释一下,为了抓住后一个网页我应该采取什么步骤,我特别感兴趣的是如何从Chrome中找到正确的信息( - > Inspect - > Network - > XHR)和如何从后一个信息构建data变量(我输入requests)?

1 个答案:

答案 0 :(得分:0)

您可以使用dict.setdefault

Scrapy ,一个快速的高级网络爬行&抓取Python的框架。

https://github.com/scrapy/scrapy/