使用“美丽汤”刮擦多页-仅显示最后一页

时间：2018-10-09 19:48:43

标签： python web-scraping beautifulsoup scrape

我和我的朋友每年都会做一次FIFA选秀，我一直在寻找一种更新球员信息的快速方法。

我想要从不同页面的播放器中抓取一些信息。这些页面是：

https://sofifa.com/players?offset=0
https://sofifa.com/players?offset=51
https://sofifa.com/players?offset=101

这是我的代码：

import requests
from bs4 import BeautifulSoup
import re

pages = []

collection = ['0', '51', '101']
for i in collection:
    url = 'https://sofifa.com/players?offset=' + str(i)
    pages.append(url)

for item in pages:
    page = requests.get(item)
    soup = BeautifulSoup(page.text, 'html.parser')

table = soup.find('table', {'class': 'table-hover'})
tbody = table.find('tbody')
info = tbody.find_all('tr')


records = []
for infos in info:
    nome = infos.find("a", href=re.compile("/player/")).string
    numero = infos.find('img').get('id')
    idade = infos.find('div' , {'class': 'col-ae'}).string
    ponto = infos.find('div' , {'class': 'col-oa'}).string
    potencial = infos.find('div' , {'class': 'col-pt'}).string
    records.append((nome, numero, idade, ponto, potencial))

import pandas as pd
df = pd.DataFrame(records, columns=['nome', 'numero', 'idade', 'ponto', 'potencial'])

df.to_excel('jogadores.xls', index=False, encoding='utf-8')

问题是，我仅从页面“ 101”获得结果，而未分析前两个页面。我在这里到处搜索，但是没有运气。

我很困惑的是，如果我打印（汤），它将向我显示三页中的HTML！

我该如何解决？

这是HTML代码的一个示例：

<table class="table table-hover persist-area">
    <thead>...</thead>
    <tbody>         <tr>
            <td>                        <figure class="avatar">
                        <img alt="" data-src="https://cdn.sofifa.org/players/4/19/158023.png" data-srcset="https://cdn.sofifa.org/players/4/19/158023@2x.png 2x, https://cdn.sofifa.org/players/4/19/158023@3x.png 3x" src="https://cdn.sofifa.org/players/4/19/158023.png" width="48" height="48" id="158023" class="player-check loaded" srcset="https://cdn.sofifa.org/players/4/19/158023@2x.png 2x, https://cdn.sofifa.org/players/4/19/158023@3x.png 3x" data-was-processed="true"></figure></td>
            <td>
                <div class="col-name text-ellipsis rtl"><a href="/players?na=52" rel="nofollow" title="Argentina"><img alt="" src="https://cdn.sofifa.org/flags/52.png" data-src="https://cdn.sofifa.org/flags/52.png" data-srcset="https://cdn.sofifa.org/flags/52@2x.png 2x, https://cdn.sofifa.org/flags/52@3x.png 3x" class="flag loaded" style="width:23px;height:17px" srcset="https://cdn.sofifa.org/flags/52@2x.png 2x, https://cdn.sofifa.org/flags/52@3x.png 3x" data-was-processed="true"></a> <a href="/player/158023" title="Lionel Messi">L. Messi</a>                    <div class="text-ellipsis rtl"><a rel="nofollow" href="/players?pn=21"><span class="pos pos21">CF</span></a> <a rel="nofollow" href="/players?pn=23"><span class="pos pos23">RW</span></a> <a rel="nofollow" href="/players?pn=25"><span class="pos pos25">ST</span></a></div>
                </div>
            </td>                   <td class="col text-center" data-col="ae">
                    <div class="col-digit col-ae">31</div>
                    </td>                   <td class="col text-center" data-col="oa">
                    <div class="col-digit col-oa"><span class="label p94">94</span></div>
                    </td>                   <td class="col text-center" data-col="pt">
                    <div class="col-digit col-pt"><span class="label p94">94</span></div>
                    </td>           <td>
                <div class="col-name text-ellipsis rtl">                        <figure class="avatar avatar-sm transparent">
                        <img alt="" class="team loaded" data-src="https://cdn.sofifa.org/teams/2/19/light/241.png" data-srcset="https://cdn.sofifa.org/teams/2/19/light/241@2x.png 2x, https://cdn.sofifa.org/teams/2/19/light/241@3x.png 3x" src="https://cdn.sofifa.org/teams/2/19/light/241.png" width="24" height="24" srcset="https://cdn.sofifa.org/teams/2/19/light/241@2x.png 2x, https://cdn.sofifa.org/teams/2/19/light/241@3x.png 3x" data-was-processed="true">
                        </figure>
                        <a href="/team/241">FC Barcelona</a>                    <div class="subtitle text-ellipsis rtl">2004 ~ 2021</div>
                </div>
            </td><th class="gap"></th>                      <td class="col text-center" data-col="vl">
                        <div class="col-digit col-vl">€110.5M</div>
                        </td>                       <td class="col text-center" data-col="wg">
                        <div class="col-digit col-wg">€565K</div>
                        </td><th class="gap"></th>                      <td class="col text-center" data-col="tt">
                        <div class="col-digit col-tt">2195</div>
                        </td>           <td class="gap"></td>
            <td>                    <div class="col-comments text-right text-ellipsis rtl">0.8K / 32.9K                 </div></td></tr>            <tr>
            <td>                        <figure class="avatar">
                        <img alt="" data-src="https://cdn.sofifa.org/players/4/19/20801.png" data-srcset="https://cdn.sofifa.org/players/4/19/20801@2x.png 2x, https://cdn.sofifa.org/players/4/19/20801@3x.png 3x" src="https://cdn.sofifa.org/players/4/19/20801.png" width="48" height="48" id="20801" class="player-check loaded" srcset="https://cdn.sofifa.org/players/4/19/20801@2x.png 2x, https://cdn.sofifa.org/players/4/19/20801@3x.png 3x" data-was-processed="true"></figure></td>
            <td>
                <div class="col-name text-ellipsis rtl"><a href="/players?na=38" rel="nofollow" title="Portugal"><img alt="" src="https://cdn.sofifa.org/flags/38.png" data-src="https://cdn.sofifa.org/flags/38.png" data-srcset="https://cdn.sofifa.org/flags/38@2x.png 2x, https://cdn.sofifa.org/flags/38@3x.png 3x" class="flag loaded" style="width:23px;height:17px" srcset="https://cdn.sofifa.org/flags/38@2x.png 2x, https://cdn.sofifa.org/flags/38@3x.png 3x" data-was-processed="true"></a> <a href="/player/20801" title="C. Ronaldo dos Santos Aveiro">Cristiano Ronaldo</a>                 <div class="text-ellipsis rtl"><a rel="nofollow" href="/players?pn=25"><span class="pos pos25">ST</span></a> <a rel="nofollow" href="/players?pn=27"><span class="pos pos27">LW</span></a></div>
                </div>
            </td>                   <td class="col text-center" data-col="ae">
                    <div class="col-digit col-ae">33</div>
                    </td>                   <td class="col text-center" data-col="oa">
                    <div class="col-digit col-oa"><span class="label p94">94</span></div>
                    </td>                   <td class="col text-center" data-col="pt">
                    <div class="col-digit col-pt"><span class="label p94">94</span></div>
                    </td>           <td>
                <div class="col-name text-ellipsis rtl">                        <figure class="avatar avatar-sm transparent">
                        <img alt="" class="team loaded" data-src="https://cdn.sofifa.org/teams/2/19/light/45.png" data-srcset="https://cdn.sofifa.org/teams/2/19/light/45@2x.png 2x, https://cdn.sofifa.org/teams/2/19/light/45@3x.png 3x" src="https://cdn.sofifa.org/teams/2/19/light/45.png" width="24" height="24" srcset="https://cdn.sofifa.org/teams/2/19/light/45@2x.png 2x, https://cdn.sofifa.org/teams/2/19/light/45@3x.png 3x" data-was-processed="true">
                        </figure>
                        <a href="/team/45">Juventus</a>                 <div class="subtitle text-ellipsis rtl">2018 ~ 2022</div>
                </div>
            </td><th class="gap"></th>                      <td class="col text-center" data-col="vl">
                        <div class="col-digit col-vl">€77M</div>
                        </td>                       <td class="col text-center" data-col="wg">
                        <div class="col-digit col-wg">€405K</div>
                        </td><th class="gap"></th>                      <td class="col text-center" data-col="tt">
                        <div class="col-digit col-tt">2228</div>
                        </td>           <td class="gap"></td>
            <td>                    <div class="col-comments text-right text-ellipsis rtl">0.8K / 40.1K                 </div></td></tr></tbody>
</table>

1 个答案:

答案 0 :(得分：0)

这是因为G_M已经指出的缩进。请尝试这种方式：

import requests
from bs4 import BeautifulSoup
import re

pages = []
records = []

collection = ['0', '51', '101']
for i in collection:
    url = 'https://sofifa.com/players?offset=' + str(i)
    pages.append(url)

for item in pages:
    page = requests.get(item)
    soup = BeautifulSoup(page.text, 'html.parser')

    table = soup.find('table', {'class': 'table-hover'})
    tbody = table.find('tbody')
    info = tbody.find_all('tr')

    for infos in info:
        nome = infos.find("a", href=re.compile("/player/")).string
        numero = infos.find('img').get('id')
        idade = infos.find('div' , {'class': 'col-ae'}).string
        ponto = infos.find('div' , {'class': 'col-oa'}).string
        potencial = infos.find('div' , {'class': 'col-pt'}).string
        records.append((nome, numero, idade, ponto, potencial))
        print(nome,numero,idade,ponto,potencial)

import pandas as pd
df = pd.DataFrame(records, columns=['nome', 'numero', 'idade', 'ponto', 'potencial'])

df.to_excel('jogadores.xls', index=False, encoding='utf-8')
相关问题
最新问题