Question

我试图从网站上获取不同页面的所有文章标题。我注意到该网址的格式为“ http://www.shandong.gov.cn/col/col2267/index.html?uid=6820&pageNum=2”。我写了一个python程序来获取标题，它可以与第1页一起使用，但是如果我将url更改为pageNum = 2或其他数字，它仍然会从第一页打印标题。我将衷心感谢您的帮助。谢谢！

# -*- coding: utf-8 -*-
from urllib.request import Request, urlopen
from bs4 import BeautifulSoup
import re
header = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36'
req = Request('http://www.shandong.gov.cn/col/col2267/index.html?uid=6820&pageNum=2',
              headers={'User-Agent': header})

webpage = urlopen(req).read()
soup = BeautifulSoup(webpage, "html.parser")
titles = soup.findAll('div', attrs={"class":"list_a5"})
str = ''
file = open('./titles.txt', 'w')
pattern1 = re.compile(r'blank">(.*?)</a')
for title in titles:
    str = title.get_text()
    result = re.findall(pattern1,str)
    for word in result:
        print(word)
        file.write(word)
        file.write('\n')
file.close()

当我用python抓取时，对于不同的URL获得相同的输出

0 个答案: