Question

如何使用Python和BeautifulSoup将这样的表格batting gamelogs table - 转换为CSV文件？

我想要第一个标题，表示Rk，Gcar，Gtm等，而不是表格中的任何其他标题（本季节的每个月）。

这是我到目前为止的代码：

from bs4 import BeautifulSoup
from urllib2 import urlopen
import csv

def stir_the_soup():
    player_links = open('player_links.txt', 'r')
    player_ID_nums = open('player_ID_nums.txt', 'r')
    id_nums = [x.rstrip('\n') for x in player_ID_nums]
    idx = 0
    for url in player_links:
        print url
        soup = BeautifulSoup(urlopen(url), "lxml")
        p_type = ""
        if url[-12] == 'p':
            p_type = "pitching"
        elif url[-12] == 'b':
            p_type = "batting" 
        table = soup.find(lambda tag: tag.name=='table' and tag.has_attr('id') and tag['id']== (p_type + "_gamelogs"))
        header = [[val.text.encode('utf8') for val in table.find_all('thead')]]
        rows = []
        for row in table.find_all('tr'):
            rows.append([val.text.encode('utf8') for val in row.find_all('th')])
            rows.append([val.text.encode('utf8') for val in row.find_all('td')])
        with open("%s.csv" % id_nums[idx], 'wb') as f:
            writer = csv.writer(f)
            writer.writerow(header)
            writer.writerows(row for row in rows if row)
        idx += 1
    player_links.close()

if __name__ == "__main__":
    stir_the_soup()

id_nums列表包含每个播放器的所有ID号，用作单独CSV文件的名称。

对于每一行，最左边的单元格是标记，行的其余部分是标记。除了标题之外，我该如何将它放入一行？

Answer 1

import pandas as pd
from bs4 import BeautifulSoup
import urllib2

url = 'https://www.baseball-reference.com/players/gl.fcgi?id=abreuto01&t=b&year=2010'
html=urllib2.urlopen(url)

bs = BeautifulSoup(html,'lxml')

table = str(bs.find('table',{'id':'batting_gamelogs'}))

dfs = pd.read_html(table)

这使用Pandas，这对于这样的东西非常有用。它还以非常合理的格式对其进行其他操作。

https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_html.html

Answer 2

此代码为您提供了统计数据的大表，这是我认为您想要的。确保已安装lxml，beautifulsoup4和pandas。

df = pd.read_html(r'https://www.baseball-reference.com/players/gl.fcgi?id=abreuto01&t=b&year=2010')
print(df[4])

这是前5行的输出。您可能需要稍微清理它，因为我不知道您的确切结果是什么：

df[4].head(5)
    Rk  Gcar    Gtm Date    Tm  Unnamed: 5  Opp Rslt    Inngs   PA  ... CS  BA  OBP SLG OPS BOP aLI WPA RE24    Pos
0   1   66  2 (1)   Apr 6   ARI NaN SDP L,3-6   7-8 1   ... 0   1.000   1.000   1.000   2.000   9   .94 0.041   0.51    PH
1   2   67  3   Apr 7   ARI NaN SDP W,5-3   7-8 1   ... 0   .500    .500    .500    1.000   9   1.16    -0.062  -0.79   PH
2   3   68  4   Apr 9   ARI NaN PIT W,9-1   8-GF    1   ... 0   .667    .667    .667    1.333   2   .00 0.000   0.13    PH SS
3   4   69  5   Apr 10  ARI NaN PIT L,3-6   CG  4   ... 0   .500    .429    .500    .929    2   1.30    -0.040  -0.37   SS
4   5   70  7 (1)   Apr 13  ARI @   LAD L,5-9   6-6 1   ... 0   .429    .375    .429    .804    9   1.52    -0.034  -0.46   PH

选择此DataFrame中的某些列：df[4]['COLUMN_NAME_HERE'].head(5)

示例：df[4]['Gcar']

此外，如果df[4]变得烦人，您可以随时切换到另一个数据框df2=df[4]

将HTML表格转换为CSV文件

2 个答案: