如何从“ a”标签中提取链接

Question

我尝试了几种方法从以下网页提取链接，但是似乎找不到所需的链接。我正在尝试从此网页（https://www.espn.com/collegefootball/scoreboard//year/2019/seasontype/2/week/1 ）中提取“ gamecast”按钮的所有链接。我将尝试获得的第一个示例是：https://www.espn.com/college-football/game/ /gameId/401110723

当我尝试仅拉动页面上的所有链接时，我什至似乎都没有得到所需的链接，因此我很困惑我在哪里出错了。我在下面进行的一些尝试似乎并没有达到我想要的目的。我在下面尝试过的第一种方法。

import requests
import csv
from bs4 import BeautifulSoup
import pandas as pd

page = requests.get('https://www.espn.com/college-football/scoreboard/_/year/2019/seasontype/2/week/1')
soup = BeautifulSoup(page.text, 'html.parser')
# game_id = soup.find(name_='&lpos=college-football:scoreboard:gamecast')
game_id = soup.find('a',class_='button-alt sm')

这是我尝试的第二种方法。任何帮助将不胜感激。

for a in soup.find_all('a'):
if 'college-football' in a['href']:
print(link['href'])

编辑：为澄清起见，我尝试拉出所有包含gameID的链接，如示例链接中所示。

Answer 1

带有您要链接的按钮的按钮已加载javascript。请求模块不会在正在搜索的html中加载javascript。因此，您不能直接刮取按钮来找到所需的链接（没有像Selenium这样的网页模拟器）。但是，我在html中找到了json数据，其中包含链接所在的记分板数据。如果您还希望从该页面抓取更多信息（时间等），我强烈建议您浏览json数据。代码中的变量json_scoreboard。

代码

import requests, re, json
from bs4 import BeautifulSoup

r = requests.get(r'https://www.espn.com/college-football/scoreboard/_/year/2019/seasontype/2/week/1')
soup = BeautifulSoup(r.text, 'html.parser')

scripts_head = soup.find('head').find_all('script')
all_links = {}
for script in scripts_head:
    if 'window.espn.scoreboardData' in script.text:
        json_scoreboard = json.loads(re.search(r'({.*?});', script.text).group(1))
        for event in json_scoreboard['events']:
            name = event['name']
            for link in event['links']:
                if link['text'] == 'Gamecast':
                    gamecast = link['href']
            all_links[name] = gamecast

print(all_links)

输出

{'Miami Hurricanes at Florida Gators': 'http://www.espn.com/college-football/game/_/gameId/401110723', 'Georgia Tech Yellow Jackets at Clemson Tigers': 'http://www.espn.com/college-football/game/_/gameId/401111653', 'Texas State Bobcats at Texas A&M Aggies': 'http://www.espn.com/college-football/game/_/gameId/401110731', 'Utah Utes at BYU Cougars': 'http://www.espn.com/college-football/game/_/gameId/401114223', 'Florida A&M Rattlers at UCF Knights': 'http://www.espn.com/college-football/game/_/gameId/401117853', 'Tulsa Golden Hurricane at Michigan State Spartans': 'http://www.espn.com/college-football/game/_/gameId/401112212', 'Wisconsin Badgers at South Florida Bulls': 'http://www.espn.com/college-football/game/_/gameId/401117856', 'Duke Blue Devils at Alabama Crimson Tide': 'http://www.espn.com/college-football/game/_/gameId/401110720', 'Georgia Bulldogs at Vanderbilt Commodores': 'http://www.espn.com/college-football/game/_/gameId/401110732', 'Florida Atlantic Owls at Ohio State Buckeyes': 'http://www.espn.com/college-football/game/_/gameId/401112251', 'Georgia Southern Eagles at LSU Tigers': 'http://www.espn.com/college-football/game/_/gameId/401110725', 'Middle Tennessee Blue Raiders at Michigan Wolverines': 'http://www.espn.com/college-football/game/_/gameId/401112222', 'Louisiana Tech Bulldogs at Texas Longhorns': 'http://www.espn.com/college-football/game/_/gameId/401112135', 'Oregon Ducks at Auburn Tigers': 'http://www.espn.com/college-football/game/_/gameId/401110722', 'Eastern Washington Eagles at Washington Huskies': 'http://www.espn.com/college-football/game/_/gameId/401114233', 'Idaho Vandals at Penn State Nittany Lions': 'http://www.espn.com/college-football/game/_/gameId/401112257', 'Miami (OH) RedHawks at Iowa Hawkeyes': 'http://www.espn.com/college-football/game/_/gameId/401112191', 'Northern Iowa Panthers at Iowa State Cyclones': 'http://www.espn.com/college-football/game/_/gameId/401112085', 'Syracuse Orange at Liberty Flames': 'http://www.espn.com/college-football/game/_/gameId/401112434', 'New Mexico State Aggies at Washington State Cougars': 'http://www.espn.com/college-football/game/_/gameId/401114228', 'South Alabama Jaguars at Nebraska Cornhuskers': 'http://www.espn.com/college-football/game/_/gameId/401112238', 'Northwestern Wildcats at Stanford Cardinal': 'http://www.espn.com/college-football/game/_/gameId/401112245', 'Houston Cougars at Oklahoma Sooners': 'http://www.espn.com/college-football/game/_/gameId/401112114', 'Notre Dame Fighting Irish at Louisville Cardinals': 'http://www.espn.com/college-football/game/_/gameId/401112436'}