Question

我正在尝试从HTML文件中抓取数据。它看起来像这样：

from bs4 import BeautifulSoup as bs
import urllib
redditPage1 = "http://redditlist.com/sfw"
r=urllib.urlopen(redditPage1).read()
soup = bs(r)

现在我希望按照订阅者数量的顺序在列表中获取reddit版主（或称为子编辑者）。为此，我只需要查看这行代码之后的数据：

<h3 class="listing-header">Subscribers</h3>

此行之前的所有内容都无关紧要，此行之后的所有关于子订阅者的条目如下所示：

<div class="listing-item" data-target-filter="sfw" data-target-subreddit="funny">
<div class="offset-anchor" id="funny-subscribers"></div>
<span class="rank-value">1</span>
<span class="subreddit-info-panel-toggle sfw"> <div>i</div> </span>
<span class="subreddit-url">
<a class="sfw" href="http://reddit.com/r/funny" target="_blank">funny</a>
</span>
<span class="listing-stat">18,197,786</span>
</div>

我应该怎样做才能提取在此行之后而不是之前的子编辑名称？

Answer 1

尝试找到<h3 class="listing-header">Subscribers</h3>，然后获取父div，范围将限制为Subscribers div。然后找到其类为listing-item的所有div，循环它们以获取内部元素<a>的文本（名称）：

from bs4 import BeautifulSoup as bs
import urllib
redditPage1 = "http://redditlist.com/sfw"
r=urllib.urlopen(redditPage1).read()
soup = bs(r,'lxml')
for sub_div in soup.find("h3", text="Subscribers").parent.find_all('div',{ "class" : "listing-item" }):
    print(sub_div.find('a').getText())

Answer 2

为了获得理想的结果，使您的代码更具可读性，您也可以这样做。

import requests
from lxml.html import fromstring

res = requests.get("http://redditlist.com/sfw").text
root = fromstring(res)
for container in root.cssselect(".listing"):
    if container.cssselect("h3:contains('Subscribers')"):
        for subreddit in container.cssselect(".listing-item"):
            print(subreddit.attrib['data-target-subreddit'])

如果您愿意，可以使用BeautifulSoup：

import requests
from bs4 import BeautifulSoup

main_link = "http://redditlist.com/all?page={}"
for link in [main_link.format(page) for page in range(1,5)]:
    res = requests.get(link).text
    soup = BeautifulSoup(res,"lxml")
    for container in soup.select(".listing"):
        if container.select("h3")[0].text=="Subscribers":
            for subreddit in container.select(".listing-item"):
                print(subreddit['data-target-subreddit'])

Answer 3

试试这个：

for div in soup.select('.span4.listing'):
    if div.h3.text.lower()=='subscribers':
        output = [(ss.select('a.sfw')[0].text, ss.select('.listing-stat')[0].text) for ss in div.select('.listing-item')]

如何从某一行开始刮取HTML文件中的数据

3 个答案: