我想使用find_all中的BeautifulSoup来进行板球比赛的逐球数据。代码是：
import requests
from bs4 import BeautifulSoup
url = 'http://www.espncricinfo.com/series/10904/commentary/1075502/south-africa-vs-bangladesh-1st-test-bangladesh-tour-of-sa-2017-18'
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')
comment = soup.find_all('div', class_ = "over-circle")
print(len(comment))
print(comment[22])
我阅读了有关此问题的先前问题的答案，几乎所有问题都涉及使用不同的html解析器。我已经尝试过lxml, html.parser, html5lib，但以上提到的所有方法（以前的问题中大多建议使用）似乎都没有给出不同的结果。
没有球的数量显示为23，而应该更多。输出：
23
<div class="over-circle low-score" data-reactid="463"><span class="over-score" data-reactid="464">0</span></div>
答案 0 :(得分：1)
页面是动态的，因此并非全部呈现。您可以直接访问源并获取json响应，其中还包括页面总数。一旦有了页面总数，就可以使用查询参数遍历页面，并从上一页开始对其进行添加，以获取所有数据的最终输出。
我不知道您真正感兴趣的数据是什么，但是一切都在那里。我将其转换为数据框，但是您可以使用json结构执行您想要的操作：
但是它是嵌套的。 athletesInvolved列由列出的字典组成。如果需要，您仍然可以将其标准化/展平（让我知道您是否也想这样做，这很容易做到），但是显然会增加行/列的数量。
import requests
from pandas.io.json import json_normalize
url = 'https://site.web.api.espn.com/apis/site/v2/sports/cricket/10904/playbyplay'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36'}
payload = {
'contentorigin': 'espn',
'event': '1075502',
'page': '1',
'period': '4',
'section': 'cricinfo'}
# Get inital page of data, including total number of pages to iterate through
response = requests.get(url, headers=headers, params=payload).json()
pageCount = response['commentary']['pageCount']
print ('Total pages: %s\nProcessed page: 1' %(pageCount))
# Store the initial page to jsonData, iterate through the next `pageCount` pages and add that to the list for a final result
jsonData = response
for page in range(2, pageCount+1):
payload = {
'contentorigin': 'espn',
'event': '1075502',
'page': page,
'period': '4',
'section': 'cricinfo'}
response = requests.get(url, headers=headers, params=payload).json()
jsonData['commentary']['items'] = jsonData['commentary']['items'] + response['commentary']['items']
print ('Processed page: %s' %page)
df = json_normalize(jsonData['commentary']['items'])
输出：198行中前5行的示例
print (df.head(5).to_string())
athletesInvolved awayScore batsman.athlete.displayName batsman.athlete.fullName batsman.athlete.id batsman.athlete.name batsman.athlete.shortName batsman.faced batsman.fours batsman.runs batsman.sixes batsman.team.abbreviation batsman.team.displayName batsman.team.id batsman.team.name batsman.totalRuns bowler.athlete.displayName bowler.athlete.fullName bowler.athlete.id bowler.athlete.name bowler.athlete.shortName bowler.balls bowler.conceded bowler.maidens bowler.overs bowler.team.abbreviation bowler.team.displayName bowler.team.id bowler.team.name bowler.wickets clock date dismissal.batsman.athlete.displayName dismissal.batsman.athlete.fullName dismissal.batsman.athlete.id dismissal.batsman.athlete.name dismissal.batsman.athlete.shortName dismissal.bowled dismissal.bowler.athlete.displayName dismissal.bowler.athlete.fullName dismissal.bowler.athlete.id dismissal.bowler.athlete.name dismissal.bowler.athlete.shortName dismissal.dismissal dismissal.minutes dismissal.retiredText dismissal.text dismissal.type homeScore id innings.ballLimit innings.balls innings.byes innings.day innings.fallOfWickets innings.id innings.legByes innings.noBalls innings.number innings.remainingBalls innings.remainingOvers innings.remainingRuns innings.runRate innings.runs innings.session innings.target innings.totalRuns innings.wickets innings.wides mediaId otherBatsman.athlete.displayName otherBatsman.athlete.fullName otherBatsman.athlete.id otherBatsman.athlete.name otherBatsman.athlete.shortName otherBatsman.faced otherBatsman.fours otherBatsman.runs otherBatsman.sixes otherBatsman.team.abbreviation otherBatsman.team.displayName otherBatsman.team.id otherBatsman.team.name otherBatsman.totalRuns otherBowler.athlete.displayName otherBowler.athlete.fullName otherBowler.athlete.id otherBowler.athlete.name otherBowler.athlete.shortName otherBowler.balls otherBowler.conceded otherBowler.maidens otherBowler.overs otherBowler.team.abbreviation otherBowler.team.displayName otherBowler.team.id otherBowler.team.name otherBowler.wickets over.actual over.ball over.balls over.byes over.complete over.legByes over.limit over.maiden over.noBall over.number over.overs over.runs over.unique over.wickets over.wide period periodText playType.description playType.id postText preText scoreValue sequence shortText speedKPH speedMPH team.abbreviation team.displayName team.id team.name text
0 [{'id': '56194', 'name': 'Tamim Iqbal', 'short... 0 Tamim Iqbal Tamim Iqbal Khan 56194 Tamim Iqbal Tamim 1 0 0 0 BDESH Bangladesh 25 Bangladesh 0 Morne Morkel Morne Morkel 46538 Morne Morkel Morkel 1 0 0 0.1 SA South Africa 3 South Africa 0 00:00 2017-09-28T10:00 Tamim Iqbal Tamim Iqbal Khan 56194 Tamim Iqbal Tamim False Morne Morkel Morne Morkel 46538 Morne Morkel Morkel False 0 NaN 0 410 0 1 0 4 0 199062 0 0 4 0 0.0 424 0.0 0 2 424 0 0 0 0 Imrul Kayes Imrul Kayes 280734 Imrul Kayes Imrul 0 0 0 0 BDESH Bangladesh 25 Bangladesh 0 NaN NaN NaN NaN NaN 0 0 0 NaN NaN NaN NaN NaN 0 0.1 1 6 0 False 0 0.0 1 0 1 0.1 0 0.01 2 0 4 4th innings no run 2 <b>2.25pm</b> South Africa gather into a huddl... 0 400001 Morkel to Tamim Iqbal, no run 138.452 86.030 BDESH Bangladesh 25 Bangladesh fullish length ball, angled in from wide of th...
1 [{'id': '56194', 'name': 'Tamim Iqbal', 'short... 0 Tamim Iqbal Tamim Iqbal Khan 56194 Tamim Iqbal Tamim 2 0 0 0 BDESH Bangladesh 25 Bangladesh 0 Morne Morkel Morne Morkel 46538 Morne Morkel Morkel 2 0 0 0.2 SA South Africa 3 South Africa 0 00:00 2017-09-28T10:00 Tamim Iqbal Tamim Iqbal Khan 56194 Tamim Iqbal Tamim False Morne Morkel Morne Morkel 46538 Morne Morkel Morkel False 0 NaN 0 420 0 2 0 4 0 199062 0 0 4 0 0.0 424 0.0 0 2 424 0 0 0 0 Imrul Kayes Imrul Kayes 280734 Imrul Kayes Imrul 0 0 0 0 BDESH Bangladesh 25 Bangladesh 0 NaN NaN NaN NaN NaN 0 0 0 NaN NaN NaN NaN NaN 0 0.2 2 6 0 False 0 0.0 1 0 1 0.2 0 0.02 2 0 4 4th innings no run 2 0 400002 Morkel to Tamim Iqbal, no run 135.891 84.439 BDESH Bangladesh 25 Bangladesh length ball outside off, Tamim stands tall and...
2 [{'id': '56194', 'name': 'Tamim Iqbal', 'short... 0 Tamim Iqbal Tamim Iqbal Khan 56194 Tamim Iqbal Tamim 3 0 0 0 BDESH Bangladesh 25 Bangladesh 0 Morne Morkel Morne Morkel 46538 Morne Morkel Morkel 3 0 0 0.3 SA South Africa 3 South Africa 0 00:00 2017-09-28T10:00 Tamim Iqbal Tamim Iqbal Khan 56194 Tamim Iqbal Tamim False Morne Morkel Morne Morkel 46538 Morne Morkel Morkel False 0 NaN 0 430 0 3 0 4 0 199062 0 0 4 0 0.0 424 0.0 0 2 424 0 0 0 0 Imrul Kayes Imrul Kayes 280734 Imrul Kayes Imrul 0 0 0 0 BDESH Bangladesh 25 Bangladesh 0 NaN NaN NaN NaN NaN 0 0 0 NaN NaN NaN NaN NaN 0 0.3 3 6 0 False 0 0.0 1 0 1 0.3 0 0.03 2 0 4 4th innings no run 2 Zahi: "The six went for four? Last ball needs ... 0 400003 Morkel to Tamim Iqbal, no run 140.489 87.296 BDESH Bangladesh 25 Bangladesh fullish, comes into Tamim who flicks it to mid...
3 [{'id': '56194', 'name': 'Tamim Iqbal', 'short... 0 Tamim Iqbal Tamim Iqbal Khan 56194 Tamim Iqbal Tamim 4 0 0 0 BDESH Bangladesh 25 Bangladesh 0 Morne Morkel Morne Morkel 46538 Morne Morkel Morkel 4 0 0 0.4 SA South Africa 3 South Africa 1 00:00 2017-09-28T10:00 Tamim Iqbal Tamim Iqbal Khan 56194 Tamim Iqbal Tamim True Morne Morkel Morne Morkel 46538 Morne Morkel Morkel True 2 Tamim Iqbal b Morkel 0 (2m 4b 0x4 0x6) SR: 0.00 bowled 0 440 0 4 0 4 1 199062 0 0 4 0 0.0 424 0.0 0 2 424 0 1 0 0 Imrul Kayes Imrul Kayes 280734 Imrul Kayes Imrul 0 0 0 0 BDESH Bangladesh 25 Bangladesh 0 NaN NaN NaN NaN NaN 0 0 0 NaN NaN NaN NaN NaN 0 0.4 4 6 0 False 0 0.0 1 0 1 0.4 0 0.04 2 0 4 4th innings out 9 0 400004 Morkel to Tamim Iqbal, OUT 136.028 84.524 BDESH Bangladesh 25 Bangladesh bowled him! Morkel strikes first over the chas...
4 [{'id': '373696', 'name': 'Mominul Haque', 'sh... 0 Mominul Haque Mominul Haque 373696 Mominul Haque Mominul 1 0 0 0 BDESH Bangladesh 25 Bangladesh 0 Morne Morkel Morne Morkel 46538 Morne Morkel Morkel 5 0 0 0.5 SA South Africa 3 South Africa 1 00:00 2017-09-28T10:00 Mominul Haque Mominul Haque 373696 Mominul Haque Mominul False Morne Morkel Morne Morkel 46538 Morne Morkel Morkel False 0 NaN 0 450 0 5 0 4 0 199062 0 0 4 0 0.0 424 0.0 0 2 424 0 1 0 0 Imrul Kayes Imrul Kayes 280734 Imrul Kayes Imrul 0 0 0 0 BDESH Bangladesh 25 Bangladesh 0 NaN NaN NaN NaN NaN 0 0 0 NaN NaN NaN NaN NaN 0 0.5 5 6 0 False 0 0.0 1 0 1 0.5 0 0.05 2 0 4 4th innings no run 2 0 400005 Morkel to Mominul Haque, no run 139.982 86.981 BDESH Bangladesh 25 Bangladesh <b>huge appeal for a leg before</b>. Not out s...
答案 1 :(得分：0)
您猜对了。并非一次加载所有数据（因此，您只能看到最初加载的数据）。您可以实现其他逻辑，这些逻辑将一直循环直到程序到达最后一页。
以下是其中一个数据页面的URL：https://site.web.api.espn.com/apis/site/v2/sports/cricket/10904/playbyplay?contentorigin=espn&event=1075502&page=6&period=4§ion=cricinfo您将需要增加page参数，直到您继续获取有效数据。
如果您检查此URL的响应，则会看到这是JSON文件，其中包含其他24个项目。