Question

我想从whoscored获取所有玩家链接。问题是，BS无法找到与班级player-link的任何链接。获得这些链接的最佳方式是什么？它们存在于检查员出现的某个地方。

到目前为止，这是我的代码：

import requests
from bs4 import BeautifulSoup as BS
import re
from incapsula import IncapSession
import json

session = IncapSession()
links = "https://www.whoscored.com/Matches/1080633/Live/England-Premier-League-2016-2017-Manchester-United-Arsenal"
try:
    response = session.get(links).text
except IncapBlocked as e:
    raise
soup = BS(response, "html.parser")
player_site = soup.find("a", text = "Player Statistics")
player_link = player_site.get("href")
player_link = "https://www.whoscored.com" + player_link
try:
    response_players = session.get(player_link).text
except IncapBlocked as e:
    raise
soup_players = BS(response_players, "html.parser")
soup_players.find_all("a", class_ ="player-link")

感谢您的帮助。

修改

现在我读到我得到了不同的结果，因为页面是通过JavaScript部分加载的。我该如何解决这个问题？

Answer 1

您可能希望使用selenium。

>>> from selenium import webdriver
>>> driver = webdriver.Chrome()
>>> driver.get('https://www.whoscored.com/Matches/1080633/LiveStatistics/England-Premier-League-2016-2017-Manchester-United-Arsenal')
>>> links = driver.find_elements_by_xpath('.//a[@class="player-link"]')
>>> for link in links:
...     link.get_attribute('href')
...     
'https://www.whoscored.com/Players/79554/Show/David-de-Gea'
'https://www.whoscored.com/Players/18296/Show/Antonio-Valencia'
'https://www.whoscored.com/Players/70050/Show/Marcos-Rojo'
'https://www.whoscored.com/Players/81726/Show/Phil-Jones'
'https://www.whoscored.com/Players/23220/Show/Matteo-Darmian'
'https://www.whoscored.com/Players/97752/Show/Paul-Pogba'
'https://www.whoscored.com/Players/71174/Show/Ander-Herrera'
'https://www.whoscored.com/Players/2115/Show/Michael-Carrick'
'https://www.whoscored.com/Players/25363/Show/Juan-Mata'
'https://www.whoscored.com/Players/300299/Show/Marcus-Rashford'
'https://www.whoscored.com/Players/122366/Show/Anthony-Martial'
'https://www.whoscored.com/Players/29544/Show/Morgan-Schneiderlin'
'https://www.whoscored.com/Players/70033/Show/Daley-Blind'
'https://www.whoscored.com/Players/3859/Show/Wayne-Rooney'
'https://www.whoscored.com/Players/33831/Show/Sergio-Romero'
'https://www.whoscored.com/Players/110154/Show/Memphis-Depay'
'https://www.whoscored.com/Players/109000/Show/Jesse-Lingard'
'https://www.whoscored.com/Players/8166/Show/Ashley-Young'
'https://www.whoscored.com/Players/6775/Show/Petr-Cech'
'https://www.whoscored.com/Players/91879/Show/Carl-Jenkinson'
'https://www.whoscored.com/Players/30051/Show/Laurent-Koscielny'
'https://www.whoscored.com/Players/80921/Show/Shkodran-Mustafi'
'https://www.whoscored.com/Players/23072/Show/Nacho-Monreal'
'https://www.whoscored.com/Players/125209/Show/Mohamed-Elneny'
'https://www.whoscored.com/Players/69738/Show/Francis-Coquelin'
'https://www.whoscored.com/Players/13796/Show/Theo-Walcott'
'https://www.whoscored.com/Players/13756/Show/Mesut-%C3%96zil'
'https://www.whoscored.com/Players/26820/Show/Aaron-Ramsey'
'https://www.whoscored.com/Players/25244/Show/Alexis-S%C3%A1nchez'
'https://www.whoscored.com/Players/24444/Show/Olivier-Giroud'
'https://www.whoscored.com/Players/89401/Show/Granit-Xhaka'
'https://www.whoscored.com/Players/84146/Show/Alex-Oxlade-Chamberlain'
'https://www.whoscored.com/Players/76810/Show/Gabriel'
'https://www.whoscored.com/Players/36745/Show/David-Ospina'
'https://www.whoscored.com/Players/136824/Show/Alex-Iwobi'
'https://www.whoscored.com/Players/27550/Show/Kieran-Gibbs'

使用BeautifulSoup进行抓取时链接丢失

1 个答案: