嗨,我正在学习有关从网站(click on link )抓取数据的问题,但我遇到了这个问题,我正在尝试从网站的下一页获取数据(我已经有数据了从第一页开始),当我尝试从第二页请求数据时,我又从第一页获得了数据,希望有人可以帮忙。 这是我的一些代码,在此先感谢:
import requests
from bs4 import BeautifulSoup
# url_base = "https://centuryrealestate.c21.com/real-estate/chicago-il-60610/LZ60610/#s="
def get_soups(url_base, int_pages):
"""
:param url_base: link, which we will request
:param int_pages: int, how many pages we will look
:return: list whit the soup of int_pages from url_base
"""
lst_soups = []
for s in range(int_pages):
url_aux = url_base + str(s*10)
r = requests.get(url_aux, headers={'User-agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0'})
c = r.content
soup = BeautifulSoup(c, "html.parser")
lst_soups.append(soup)
return lst_soups