Question

不确定为什么它不起作用:(我可以从此页面中拉出其他表，只是不能从该表中拉出。

import requests
from bs4 import BeautifulSoup as soup

url = requests.get("https://www.basketball-reference.com/teams/BOS/2018.html",
                   headers={'User-Agent': 'Mozilla/5.0'})

page = soup(url.content, 'html')
table = page.find('table', id='team_and_opponent')
print(table)

感谢帮助。

Answer 1

页面是动态的。因此，在这种情况下，您有2个选择。

旁注：如果看到<table>标签，请不要使用BeautifulSoup，大熊猫可以通过使用pd.read_html()来为您完成这项工作（实际上是在后台使用bs4）

1）使用硒首先呈现页面，然后您可以使用BeautifulSoup抽出<table>标签

2）这些表在html的注释标记内。您可以使用BeautifulSoup提取评论，然后仅用'table'抓取评论。

我选择了选项2。

import requests
from bs4 import BeautifulSoup
from bs4 import Comment
import pandas as pd


url = 'https://www.basketball-reference.com/teams/BOS/2018.html'
response = requests.get(url)

soup = BeautifulSoup(response.text, 'html.parser')

comments = soup.find_all(string=lambda text: isinstance(text, Comment))

tables = []
for each in comments:
    if 'table' in each:
        try:
            tables.append(pd.read_html(each)[0])
        except:
            continue

我不知道您想要哪个特定表，但是它们在tables列表中

*输出：**

print (tables[1])
   Unnamed: 0     G     MP     FG    FGA  ...    STL     BLK   TOV     PF    PTS
0        Team  82.0  19805   3141   6975  ...    604     373  1149   1618   8529
1      Team/G   NaN  241.5   38.3   85.1  ...    7.4     4.5  14.0   19.7  104.0
2     Lg Rank   NaN     12     25     25  ...     23      18    15     17     20
3   Year/Year   NaN   0.3%  -0.9%  -0.0%  ...  -2.1%    9.7%  5.6%  -4.0%  -3.7%
4    Opponent  82.0  19805   3066   6973  ...    594     364  1159   1571   8235
5  Opponent/G   NaN  241.5   37.4   85.0  ...    7.2     4.4  14.1   19.2  100.4
6     Lg Rank   NaN     12      3     12  ...      7       6    19      9      3
7   Year/Year   NaN   0.3%  -3.2%  -0.9%  ...  -4.7%  -14.4%  1.6%  -5.6%  -4.7%

[8 rows x 24 columns]

或

print (tables[18])
    Rk          Unnamed: 1       Salary
0    1      Gordon Hayward  $29,727,900
1    2          Al Horford  $27,734,405
2    3        Kyrie Irving  $18,868,625
3    4        Jayson Tatum   $5,645,400
4    5         Greg Monroe   $5,000,000
5    6       Marcus Morris   $5,000,000
6    7        Jaylen Brown   $4,956,480
7    8        Marcus Smart   $4,538,020
8    9         Aron Baynes   $4,328,000
9   10  Guerschon Yabusele   $2,247,480
10  11        Terry Rozier   $1,988,520
11  12        Shane Larkin   $1,471,382
12  13        Semi Ojeleye   $1,291,892
13  14         Abdel Nader   $1,167,333
14  15        Daniel Theis     $815,615
15  16   Demetrius Jackson      $92,858
16  17        Jarell Eddie      $83,129
17  18        Xavier Silas      $74,159
18  19     Jonathan Gibson      $44,495
19  20         Jabari Bird           $0
20  21        Kadeem Allen           $0

Answer 2

该页面中没有ID为table的{{1}}。而是有一个带有此ID的span标签。您可以通过更改ID获得结果。

Answer 3

此数据应动态加载（如JavaScript）。

您应该在这里Web-scraping JavaScript page with Python

为此，您可以使用支持Javascript的Selenium或html_requests

Answer 4

import requests
import bs4
url = requests.get("https://www.basketball-reference.com/teams/BOS/2018.html",
                   headers={'User-Agent': 'Mozilla/5.0'})
soup=bs4.BeautifulSoup(url.text,"lxml")
page=soup.select(".table_outer_container")
for i in page:
    print(i.text)

您将获得所需的输出

bs4按ID查找表，返回“无”

4 个答案: