Question

我使用Python来抓取this webpage。但是当你点击上一个按钮时我无法取出这些物品。我想过使用Selenium来实现这一目标，但我没有成功让它变得无头。

使用下面的代码我刮掉匹配链接：

import urllib2
import re

site_url = 'http://us.soccerway.com'
national_league_div_sub_matches_url = 'http://us.soccerway.com/national/england/premier-league/20132014/regular-season/r21322/'
national_league_div_sub_matches_url_source = urllib2.urlopen(national_league_div_sub_matches_url).read()
match_links = re.findall('(/matches/[0-9][0-9][0-9][0-9]/.*?ICID.*?)">', national_league_div_sub_matches_url_source)
match_links = map(lambda x: ''.join([site_url, x]), match_links)
for x in match_links:
    print x

Answer 1

当您在浏览器中点击previous时，javascript会调用此url长来从服务器获取JSON数据 - 所以也一样。

import requests, json

url = 'http://us.soccerway.com/a/block_competition_matches_summary?block_id=page_competition_1_block_competition_matches_summary_6&callback_params=%7B%22page%22%3A0%2C%22bookmaker_urls%22%3A%7B%2213%22%3A%5B%7B%22link%22%3A%22http%3A%2F%2Fwww.bet365.com%2Fhome%2F%3Faffiliate%3D365_308124%22%2C%22name%22%3A%22Bet%20365%22%7D%5D%7D%2C%22block_service_id%22%3A%22competition_summary_block_competitionmatchessummary%22%2C%22round_id%22%3A21322%2C%22outgroup%22%3Afalse%2C%22view%22%3A2%7D&action=changePage&params=%7B%22page%22%3A-1%7D'

r = requests.get(url)

#print r.text

data = json.loads(r.text)

print data

现在dict中有data，因此您必须找到所需内容。

可能每次点击previous进行broswer更改url，因此如果您想获取较旧的数据，则必须执行相同的操作。

如何使用python在网页中刮取表格？

1 个答案: