XHR请求URL在尝试解析其内容时不存在

时间:2014-09-03 22:32:09

标签: python web-scraping xmlhttprequest python-requests

在使用Scrapy构建我的问题的完整解决方案之前,我发布了一个简单版本的我想要做的事情:

import requests

url = 'http://www.whoscored.com/stageplayerstatfeed/?field=1&isAscending=false&orderBy=Rating&playerId=-1&stageId=9155&teamId=32"'

params = {'d': date.strftime('%Y%m'), 'isAggregate': 'false'}
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36'}

response = requests.get(url, params=params, headers=headers)

fixtures = response.body
#fixtures = literal_eval(response.content)
print fixtures 

此代码表示上述网址不存在。该URL与您从此页面主表的“总体”标签切换到“主页”标签时提交的XHR请求相关:

http://www.whoscored.com/Teams/32/

如果您在Google Developer Tools的控制台中激活XHR日志记录,则可以以字典的形式(这是预期的格式)查看XHR请求和从服务器发送的响应。

有谁能告诉我为什么上面的代码没有返回我期望看到的数据?

由于

1 个答案:

答案 0 :(得分:9)

你有几个问题:

  • 网址应为http://www.whoscored.com/stageplayerstatfeed
  • 错误的GET参数
  • 缺少重要的标题
  • 您需要response.json(),而不是response.body

固定版本:

import requests

url = 'http://www.whoscored.com/stageplayerstatfeed'
params = {
    'field': '1',
    'isAscending': 'false',
    'orderBy': 'Rating',
    'playerId': '-1',
    'stageId': '9155',
    'teamId': '32'
}
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36',
           'X-Requested-With': 'XMLHttpRequest',
           'Host': 'www.whoscored.com',
           'Referer': 'http://www.whoscored.com/Teams/32/'}

response = requests.get(url, params=params, headers=headers)

fixtures = response.json()
print fixtures

打印:

[
    {
        u'AccurateCrosses': 0,
        u'AccurateLongBalls': 10,
        u'AccuratePasses': 89,
        u'AccurateThroughBalls': 0,
        u'AerialLost': 2,
        u'AerialWon': 4,
        ...
    },
    ...
]