Question

所以我写了这段代码来从网站上抓取数据。

import requests
from bs4 import BeautifulSoup
import re

page = requests.get('https://sofifa.com/shortlist/32931')
soup = BeautifulSoup(page.text, 'html.parser')
dados = soup.find_all('a', href=re.compile("/player/"))
capa = soup.find('article')
capa1 = capa.find('div' , {'class': 'card card-border mb-2 fixed-width'})
time = capa1.find('div' , {'class': 'card-title h5'}).string

records = []
for nomes in dados:
    nome = nomes.string
    records.append((nome))

import pandas as pd
from openpyxl import load_workbook

book = load_workbook('Tabela Pipoco 2019.xlsx')
writer = pd.ExcelWriter('Tabela Pipoco 2019.xlsx', engine='openpyxl')
writer.book = book
writer.sheets = dict((ws.title, ws) for ws in book.worksheets)
df = pd.DataFrame(records, columns=[time])
df.to_excel(writer, "Times", index=False, encoding='utf-8', startcol=0)
writer.save()

问题是我想用10个不同的页面来制作！由于缺乏知识，我在jupyter笔记本上编写了10种不同的代码，并全部运行。

唯一更改代码的两行：

page = requests.get('https://sofifa.com/shortlist/32931')

和

df.to_excel(writer, "Times", index=False, encoding='utf-8', startcol=0)

因此，在第一行中，我们将结束号更改为特定的数字（32931、32882、32589），依此类推。

第二行，startcol=的变化是（第一页为0，第二页为3，第三页为6，依此类推）

我如何在单个代码上放置细化？

谢谢大家

Answer 1

相同的代码编写3次后，编写一个函数（c）David Robinson

您可以使用带有两个参数的函数包装代码：url和startcol，然后在循环中为不同的输入调用此函数。例如：

# Define a function
def your_func_to_avoid_writing_the_same_code_ten_times(url, col):
    page = requests.get(url)  
    # ... missed code for better formatting
    df.to_excel(writer, "Times", index=False, encoding='utf-8', startcol=col)
    writer.save()

url_list = ['https://sofifa.com/shortlist/32931', 'https://sofifa.com/shortlist/32882'] # And so on

# Initialize columns counter
col = 0
# Call the function and update the column
for i in range(len(url_list)): 
    print(your_func_to_avoid_writing_the_same_code_ten_times(url_list[i], col))
    col += 3

Answer 2

您可以列出页面，然后使用索引为每个页面bu分配一个变量。例如：

page_list = ['https://sofifa.com/shortlist/32931', 'https://sofifa.com/shortlist/32941',            'https://sofifa.com/shortlist/32931']
page1 = page_list[0]
page2 = page_list[1]
page3 = page_list[2]

您可以对pdf文件进行同样的处理以使其成为excel

Beautifulsoup和Panda-帮助修改多页代码

2 个答案: