Question

下面是一个网络抓取工具，可以成功地从团队的website中提取花名册信息并将其导出到CSV文件中。如您所见，每个团队网站都有类似的网址格式。

http://m.redsox.mlb.com/roster/
http://m.yankees.mlb.com/roster/

我正在尝试创建一个循环，该循环将遍历每个团队的网站，抓取每个球员的花名册信息，并将其写入CSV文件。在代码的开头，我创建了一个团队名称字典，并将其格式化为url以请求页面。这种策略有效，但是，代码仅在字典中列出的最后一页中循环。有谁知道如何更改此代码，以使其遍历team_list词典中的所有页面？预先感谢！

import requests
import csv
from bs4 import BeautifulSoup

team_list={'yankees','redsox'}

for team in team_list:
    page = requests.get('http://m.{}.mlb.com/roster/'.format(team))
    soup = BeautifulSoup(page.text, 'html.parser')

    soup.find(class_='nav-tabset-container').decompose()
    soup.find(class_='column secondary span-5 right').decompose()

    roster = soup.find(class_='layout layout-roster')
    names = [n.contents[0] for n in roster.find_all('a')]
    ids = [n['href'].split('/')[2] for n in roster.find_all('a')]
    number = [n.contents[0] for n in roster.find_all('td', index='0')]
    handedness = [n.contents[0] for n in roster.find_all('td', index='3')]
    height = [n.contents[0] for n in roster.find_all('td', index='4')]
    weight = [n.contents[0] for n in roster.find_all('td', index='5')]
    DOB = [n.contents[0] for n in roster.find_all('td', index='6')]
    team = [soup.find('meta',property='og:site_name')['content']] * len(names)

    with open('MLB_Active_Roster.csv', 'w', newline='') as fp:
        f = csv.writer(fp)
        f.writerow(['Name','ID','Number','Hand','Height','Weight','DOB','Team'])
        f.writerows(zip(names, ids, number, handedness, height, weight, DOB, team))

Answer 1

我相信，通过将您的词典替换为列表，您应该可以解决此问题：

import requests
import csv
import pandas as pd

from bs4 import BeautifulSoup

team_list=['yankees','redsox']
output = []

for team in team_list:
    page = requests.get('http://m.{}.mlb.com/roster/'.format(team))
    soup = BeautifulSoup(page.text, 'html.parser')

    soup.find(class_='nav-tabset-container').decompose()
    soup.find(class_='column secondary span-5 right').decompose()

    roster = soup.find(class_='layout layout-roster')
    names = [n.contents[0] for n in roster.find_all('a')]
    ids = [n['href'].split('/')[2] for n in roster.find_all('a')]
    number = [n.contents[0] for n in roster.find_all('td', index='0')]
    handedness = [n.contents[0] for n in roster.find_all('td', index='3')]
    height = [n.contents[0] for n in roster.find_all('td', index='4')]
    weight = [n.contents[0] for n in roster.find_all('td', index='5')]
    DOB = [n.contents[0] for n in roster.find_all('td', index='6')]
    team = [soup.find('meta',property='og:site_name')['content']] * len(names)

    output.append([names, ids, number, handedness, height, weight, DOB, team])

pd.DataFrame(data=output, columns=['Name','ID','Number','Hand','Height','Weight','DOB','Team']).tocsv('csvfilename.csv')

使用漂亮的汤将剪贴的数据循环到网站的不同页面

1 个答案: