Question

我正在尝试抓取网站的文本。但这只爬了12篇文章。我不知道为什么会这样。我想知道是否要爬网其他页面，该怎么办？

import requests
from bs4 import BeautifulSoup

x = int(input("start page:"))
while x < int(input("end page:")):
    x = x + 1
    url = "https://www.mmtimes.com/national-news.html?page=" + str(x)
    result = requests.get(url)
    bs_obj = BeautifulSoup(result.content, "html.parser")
    content = bs_obj.find("div", {"class": "msp-three-col"})
    read_more = content.findAll("div", {"class": "read-more"})

    for item in read_more:
        atag = item.find('a')
        link = "https://www.mmtimes.com" + atag["href"]
        linkResult = requests.get(link)
        subpage = BeautifulSoup(linkResult.content, "html.parser")
        fnresult = subpage.find("div", {"class": "field-item even"})
        print(fnresult.text)
    print("Total "+str(len(read_more))+" articles"))

Answer 1

检查以下代码，我进行了一些更改。这将产生所需的输出。

import requests
from bs4 import BeautifulSoup

x = int(input("start page:"))
y = input("end page:")

article_count = 0
while x <= int(y):
    url = "https://www.mmtimes.com/national-news.html?page=" + str(x)
    result = requests.get(url)
    bs_obj = BeautifulSoup(result.content, "html.parser")
    content = bs_obj.find("div", {"class": "msp-three-col"})
    read_more = content.findAll("div", {"class": "read-more"})

    for item in read_more:
        atag = item.find('a')
        link = "https://www.mmtimes.com" + atag["href"]
        linkResult = requests.get(link)
        subpage = BeautifulSoup(linkResult.content, "html.parser")
        fnresult = subpage.find("div", {"class": "field-item even"})
        print(fnresult.text)
    article_count += len(read_more)
    print("Total "+str(article_count)+" articles")
    x += 1

如何抓取所有页面？

1 个答案: