Question

当前正在尝试循环以下网页抓取...

我目前的问题是，我只能从表（我的表HTML放在下面）中获得第一个足球运动员，而不能让10名球员全部参加比赛，我立即想到的是循环不是没有工作，我不确定我要去哪里。我正在使用BeautifulSoup方法收集数据。

TD; DR我的错误是我的CSV文件中只有1个播放器，而不是HTML中的10个播放器

Python代码

 from urllib.request import urlopen as uReq
 from urllib.request import Request
 from bs4 import BeautifulSoup as soup

 my_url = "https://www.fctables.com/teams/stoke-194901/"

 #opening up connection , grabbing page
 uClient = uReq(my_url)
 page_html = uClient.read()
 uClient.close()

 #html parsing
 page_soup = soup(page_html, "html.parser")

 topScorers = page_soup.findAll("table",{"class":"table table-striped table-bordered table-hover stage-table table-condensed top_scores"})


 filename = "stokeGoals.csv"
 f = open(filename, "w")

 headers = "player, goal_scored, average_goal"

 f.write(headers)

 for topScorer in topScorers:
 #top 10 players who scored
 player = topScorer.a["title"]

#top 10 goalscorers for the team
goalpp = topScorer.findAll("div", {"class": "progress"})

#average goal per game
avg = topScorer.findAll("div", {"class": "label label-primary"})
avgpp = avg[0].text.strip()


print("player: " + player)
print("goal_scored: " + goalpp)
print("AVG: "+ avgpp)

f.write(player + "," +goalpp.replace("," , "|")+ "," + avgpp +"\n")

f.close()

HTML代码（我要从中检索数据的表/网站）

 <table class="table table-striped table-bordered table-hover stage-table table-condensed top_scores">
<thead>
    <tr>
        <th>#</th>
        <th class="tl">Player</th>
        <th data-toggle="tooltip" title="Goals scores by player / Goals scores by his team">goals</th>
        <th data-toggle="tooltip" title="Average goals">
            Avg
        </th>
    </tr>
</thead>
<tbody>
    <tr>
        <td>1</td>
        <td class="tl psh" data-id="212996">
            <img alt="Benik Afobe" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/s4/s4glg58a2350823d58/benik-afobe.png" width="20" /> <a href="/players/benik_afobe-212996/" title="Benik Afobe">Afobe</a>
            <div class="slider">
                <div class="inner"></div>
            </div>
        </td>
        <td width="30%">
            <div class="progress">
                <div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 100%;">
                    <span class="goal_p">6</span>
                </div>
            </div>
        </td>
        <td>
            <div class="label label-primary">0.4</div>
        </td>
    </tr>
    <tr>
        <td>2</td>
        <td class="tl psh" data-id="320050">
            <img alt="Thomas Ince" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/t5/t5ni157c703a92110b/thomas-ince.jpg" width="20" /> <a href="/players/thomas_ince-320050/" title="Thomas Ince">Ince</a>
            <div class="slider">
                <div class="inner"></div>
            </div>
        </td>
        <td width="30%">
            <div class="progress">
                <div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 83.333333333333%;">
                    <span class="goal_p">5</span>
                </div>
            </div>
        </td>
        <td>
            <div class="label label-primary">0.6</div>
        </td>
    </tr>
    <tr>
        <td>3</td>
        <td class="tl psh" data-id="308648">
            <img alt="Saido Berahino" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/po/poyhu58a234e0da106/saido-berahino.png" width="20" /> <a href="/players/saido_berahino-308648/" title="Saido Berahino">Berahino</a>
            <div class="slider">
                <div class="inner"></div>
            </div>
        </td>
        <td width="30%">
            <div class="progress">
                <div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 66.666666666667%;">
                    <span class="goal_p">4</span>
                </div>
            </div>
        </td>
        <td>
            <div class="label label-primary">0.3</div>
        </td>
    </tr>
    <tr>
        <td>4</td>
        <td class="tl psh" data-id="257340">
            <img alt="Joe Allen" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/6w/6w45558a234deae78e/joe-allen.png" width="20" /> <a href="/players/joe_allen-257340/" title="Joe Allen">Allen</a>
            <div class="slider">
                <div class="inner"></div>
            </div>
        </td>
        <td width="30%">
            <div class="progress">
                <div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 50%;">
                    <span class="goal_p">3</span>
                </div>
            </div>
        </td>
        <td>
            <div class="label label-primary">0.4</div>
        </td>
    </tr>
    <tr>
        <td>5</td>
        <td class="tl psh" data-id="234407">
            <img alt="Erik Pieters" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/et/et08558a234dd63b68/erik-pieters.png" width="20" /> <a href="/players/erik_pieters-234407/" title="Erik Pieters">Pieters</a>
            <div class="slider">
                <div class="inner"></div>
            </div>
        </td>
        <td width="30%">
            <div class="progress">
                <div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 50%;">
                    <span class="goal_p">3</span>
                </div>
            </div>
        </td>
        <td>
            <div class="label label-primary">0.4</div>
        </td>
    </tr>
    <tr>
        <td>6</td>
        <td class="tl psh" data-id="299368">
            <img alt="Peter Crouch" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/qp/qptn558a234df86f1f/peter-crouch.png" width="20" /> <a href="/players/peter_crouch-299368/" title="Peter Crouch">Crouch</a>
            <div class="slider">
                <div class="inner"></div>
            </div>
        </td>
        <td width="30%">
            <div class="progress">
                <div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 33.333333333333%;">
                    <span class="goal_p">2</span>
                </div>
            </div>
        </td>
        <td>
            <div class="label label-primary">0.3</div>
        </td>
    </tr>
    <tr>
        <td>7</td>
        <td class="tl psh" data-id="214479">
            <img alt="Bojan Krkic" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/pl/pleyv57eaedf0afeac/bojan-krkic.jpg" width="20" /> <a href="/players/bojan_krkic-214479/" title="Bojan Krkic">Krkic</a>
            <div class="slider">
                <div class="inner"></div>
            </div>
        </td>
        <td width="30%">
            <div class="progress">
                <div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 33.333333333333%;">
                    <span class="goal_p">2</span>
                </div>
            </div>
        </td>
        <td>
            <div class="label label-primary">0.4</div>
        </td>
    </tr>
    <tr>
        <td>8</td>
        <td class="tl psh" data-id="253114">
            <img alt="James McClean" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/gb/gbjmm58a234f55a560/james-mcclean.png" width="20" /> <a href="/players/james_mcclean-253114/" title="James McClean">McClean</a>
            <div class="slider">
                <div class="inner"></div>
            </div>
        </td>
        <td width="30%">
            <div class="progress">
                <div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 16.666666666667%;">
                    <span class="goal_p">1</span>
                </div>
            </div>
        </td>
        <td>
            <div class="label label-primary">0.1</div>
        </td>
    </tr>
    <tr>
        <td>9</td>
        <td class="tl psh" data-id="309022">
            <img alt="Sam Clucas" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/g7/g7dig58a234cb144a3/sam-clucas.png" width="20" /> <a href="/players/sam_clucas-309022/" title="Sam Clucas">Clucas</a>
            <div class="slider">
                <div class="inner"></div>
            </div>
        </td>
        <td width="30%">
            <div class="progress">
                <div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 16.666666666667%;">
                    <span class="goal_p">1</span>
                </div>
            </div>
        </td>
        <td>
            <div class="label label-primary">0.3</div>
        </td>
    </tr>
    <tr>
        <td>10</td>
        <td class="tl psh" data-id="215724">
            <img alt="Bruno Martins Indi" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/hk/hkung58a234de0dfaa/bruno-martins-indi.png" width="20" /> <a href="/players/bruno_martins_indi-215724/" title="Bruno Martins Indi">Indi</a>
            <div class="slider">
                <div class="inner"></div>
            </div>
        </td>
        <td width="30%">
            <div class="progress">
                <div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 16.666666666667%;">
                    <span class="goal_p">1</span>
                </div>
            </div>
        </td>
        <td>
            <div class="label label-primary">0.2</div>
        </td>
    </tr>
</tbody>

Answer 1

您指定的网页通过XMLHttpRequest

加载数据

您可以直接从以下位置获取html：

https://www.fctables.com/xml/table_participant/?template_id=&season_id=52%2C38%2C88&type_home=overall&type=top_score&lang_id=2&team_id=194901&limit=10

通过上述网址，您可以获得所需的所有信息，而没有多余的html 噪声，即：

my_url = "https://www.fctables.com/xml/table_participant/?template_id=&season_id=52%2C38%2C88&type_home=overall&type=top_score&lang_id=2&team_id=194901&limit=10"

uClient = uReq(my_url)
page_html = uClient.read()
uClient.close()

page_soup = soup(page_html, "html.parser")

_names = page_soup.findAll("img",{"class":"img-circle"})
_goals = page_soup.findAll("span",{"class":"goal_p"})
_avg = page_soup.findAll("div",{"class":"label label-primary"})
x = 0
for name in _names:
    name = name['alt']
    avg = _avg[x].get_text()
    goals = _goals[x].get_text()
    print(name, avg, goals)
    x+=1

Benik Afobe 0.4 6
Thomas Ince 0.6 5
Saido Berahino 0.3 4
Joe Allen 0.4 3
Erik Pieters 0.4 3
Peter Crouch 0.3 2
Bojan Krkic 0.4 2
James McClean 0.1 1
Sam Clucas 0.3 1
Bruno Martins Indi 0.2 1

注意：

根据需要调整url的值，您可以更改top_score，type，team_id，limit等...

Answer 2

保持网址不变，您可以尝试以下操作以获取所需的结果：

import requests
from bs4 import BeautifulSoup

url = "https://www.fctables.com/teams/stoke-194901/"

res = requests.get(url)
soup = BeautifulSoup(res.text,"html.parser")
for items in soup.select(".top_scores tbody tr"):
    name = items.select_one("td a[href^='/players/']").get("title")
    goal = items.select_one("td .goal_p").text
    avrg = items.select_one("td .label-primary").text
    print(name, goal, avrg)

您应该获得的输出：

Benik Afobe 6 0.4
Thomas Ince 5 0.6
Saido Berahino 4 0.3
Joe Allen 3 0.4
Erik Pieters 3 0.4
Peter Crouch 2 0.3
Bojan Krkic 2 0.4
James McClean 1 0.1
Sam Clucas 1 0.3
Bruno Martins Indi 1 0.2

Python循环和Web废弃| BeautifulSoup

2 个答案: