我希望使用python从espn网站中抓取一些数据。
http://www.espn.co.uk/rugby/playerstats?gameId=293905&league=289234
我正在用漂亮的汤从页面中提取内容。
item = soup.findAll('span', attrs={'data-reactid': '136'})[0].text
只会显示列标题。该链接内部是未在网址中显示的数据反应性链接。一个如何导航数据反应链接?单击防御或攻击链接时,URL保持不变?
答案 0 :(得分:1)
beautifulsoup
路径看起来很困难。我认为这可能对您有用:
import requests
import json
import re
url = "http://www.espn.co.uk/rugby/playerstats?gameId=293905&league=289234"
html_doc = requests.get(url)
# not the best regex but it works. there's a lot of data.
stats = json.loads(re.search(r"window.__INITIAL_STATE__\s*=\s*({.*});",html_doc.text).group(1))
# show what we have
stats['gamePackage']['matchLineUp'].keys()
# Out[42]: dict_keys(['text', 'home', 'away', 'gameState', 'sport', 'show'])
# no idea what this sport is. a typo?
stats['gamePackage']['matchLineUp']['sport']
# Out[43]: 'rugby'
stats['gamePackage']['matchLineUp']['home']
# {'name': 'ITALY',
# 'logo': 'http://a1.espncdn.com/combiner/i?img=/i/teamlogos/rugby/teams/500/20.png&h=35&w=35',
# 'team': [
# {'id': '91554',
# 'url': 'http://en.espn.co.uk/sport/rugby/player/91554.html',
# 'name': 'Jayden Hayward',
# 'number': '15',
# 'position': 'FB',
# 'captain': False,
# 'subbed': False,
# 'homeAway': 'home',
# ...
您可以迭代,也可以进行以下操作:
for home_player in stats['gamePackage']['matchLineUp']['home']['team']:
print("{} - {}".format(home_player['name'], home_player['number']))
Jayden Hayward - 15
Tommaso Benvenuti - 14
Michele Campagnaro - 13
Tommaso Castello - 12
Luca Sperandio - 11
Tommaso Allan - 10
Tito Tebaldi - 9
Andrea Lovotti - 1
Leonardo Ghiraldini - 2
Simone Ferrari - 3
Alessandro Zanni - 4
Dean Budd - 5
Sebastian Negri - 6
Jake Polledri - 7
Braam Steyn - 8
那里还有很多其他信息,但认为这可以帮助您...