Question

我试图抓取http://www.basketball-reference.com/awards/all_league.html进行分析，我的目标如下：

0 Marc Gasol 2014-2015 2014年 1 1st Anthony Davis 2014-2015
2第一届勒布朗詹姆斯2014-2015 3 James James Harden 2014-2015 4 Stephen Curry 2014-2015 2014 5 2nd Paul Gasol 2014-2015等等

这是我到目前为止的代码，无论如何要做到这一点？任何建议/帮助非常感谢。

r = requests.get('http://www.basketball-reference.com/awards/all_league.html')
soup=BeautifulSoup(r.text.replace('&nbsp;','').replace('&gt;','').encode('ascii','ignore'),"html.parser")
all_league_data = pd.DataFrame(columns = ['year','team','player']) 


stw_list = soup.findAll('div', attrs={'class': 'stw'}) # Find all 'stw's'
for stw in stw_list:
    table = stw.find('table', attrs = {'class':'no_highlight stats_table'})
    for row in table.findAll('tr'):
        col = row.findAll('td')
        if col:
            year = col[0].find(text=True)
            team = col[2].find(text=True)
            player = col[3].find(text=True)
            all_league_data.loc[len(all_league_data)] = [team, player, year]
    all_league_data

Answer 1

看起来您的代码应该可以正常工作，但这是一个没有pandas的工作版本：

import requests
from bs4 import BeautifulSoup

r = requests.get('http://www.basketball-reference.com/awards/all_league.html')
soup=BeautifulSoup(r.text.replace('&nbsp;','').replace('&gt;','').encode('ascii','ignore'),"html.parser")
all_league_data = []

stw_list = soup.findAll('div', attrs={'class': 'stw'}) # Find all 'stw's'
for stw in stw_list:
    table = stw.find('table', attrs = {'class':'no_highlight stats_table'})
    for row in table.findAll('tr'):
        col = row.findAll('td')
        if col:
            year = col[0].find(text=True)
            team = col[2].find(text=True)
            player = col[3].find(text=True)
            all_league_data.append([team, player, year])

for i, line in enumerate(all_league_data):
    print(i, *line)

Answer 2

您已使用pandas，因此请使用read_html

import pandas as pd

all_league_data = pd.read_html('http://www.basketball-reference.com/awards/all_league.html')
print(all_league_data)

这将为您提供数据框中的所有表格数据：

  In [7]:  print(all_league_data[0].dropna().head(5))
         0    1    2                 3                   4  \
0  2014-15  NBA  1st      Marc Gasol C     Anthony Davis F   
1  2014-15  NBA  2nd       Pau Gasol C  DeMarcus Cousins C   
2  2014-15  NBA  3rd  DeAndre Jordan C        Tim Duncan F   
4  2013-14  NBA  1st     Joakim Noah C      LeBron James F   
5  2013-14  NBA  2nd   Dwight Howard C     Blake Griffin F   

                     5                6                    7  
0       LeBron James F   James Harden G      Stephen Curry G  
1  LaMarcus Aldridge F     Chris Paul G  Russell Westbrook G  
2      Blake Griffin F   Kyrie Irving G      Klay Thompson G  
4       Kevin Durant F   James Harden G         Chris Paul G  
5         Kevin Love F  Stephen Curry G        Tony Parker G

重新排列你喜欢或删除某些列是微不足道的，read_html需要一些你也可以应用的attrs，它们都在链接中。

根据另一个列值python将多个列转换为单个列

2 个答案: