Question

我正在从网上抓取多个与this one（大型击球游戏日志表）完全相同的表格，我需要数据框来忽略以赛季开始的内部标题行。

到目前为止，这是我的脚本：

from bs4 import BeautifulSoup
import pandas as pd
import csv
import urllib2

def stir_the_soup():
    player_links = open('player_links.txt', 'r')   
    player_ID_nums = open('player_ID_nums.txt', 'r')
    id_nums = [x.rstrip('\n') for x in player_ID_nums]
    idx = 0
    for url in player_links:
        #open the url and create bs object
        player_link = urllib2.urlopen(url)
        bs = BeautifulSoup(player_link, 'html5lib')

        #identify which table is needed
        table_id = ""
        if url[-12] == 'b':
            table_id = "batting"
        elif url[-12] == 'p':
            table_id = "pitching"

        #find the table and create dataframe
        table = str(bs.find('table', {'id' : (table_id + '_gamelogs')}))

        df = pd.read_html(table, header=0)
        df2 = df[0]
        df2 = df2[df2.PA != 'PA']

        #for the name of the file and file path
        file_path = '/Users/kramerbaseball/Desktop/MLB_Web_Scraping_Program/game_logs_non_concussed/'
        name_of_file = str(id_nums[idx])

        df2.to_csv(path_or_buf=(file_path + name_of_file + '.csv'), sep=',', encoding='utf-8')
        idx += 1


if __name__ == "__main__":
    stir_the_soup()

我尝试使用数据框并忽略PA == PA或HR == HR的行，但它不会删除行。任何帮助表示赞赏

Answer 1

请注意，在某些内部标题列中，值是常量。这将从您的df：

中删除中间标题

df3 = df2[df2['Gtm']!='Date']

忽略pandas数据帧中的内部标题行

1 个答案: