Python:reques.get(“某些链接”):我想查看reques.get(“相同链接,但下一页”),但给了我“某些链接”的内容

时间:2019-10-05 07:00:50

标签: python-3.x beautifulsoup python-requests

嗨,我正在学习有关从网站(click on link )抓取数据的问题,但我遇到了这个问题,我正在尝试从网站的下一页获取数据(我已经有数据了从第一页开始),当我尝试从第二页请求数据时,我又从第一页获得了数据,希望有人可以帮忙。 这是我的一些代码,在此先感谢:

import requests
from bs4 import BeautifulSoup

# url_base = "https://centuryrealestate.c21.com/real-estate/chicago-il-60610/LZ60610/#s="
def get_soups(url_base, int_pages):
    """

    :param url_base: link, which we will request
    :param int_pages: int, how many pages we will look
    :return: list whit the soup of int_pages from url_base
    """
    lst_soups = []
    for s in range(int_pages):
        url_aux = url_base + str(s*10)
        r = requests.get(url_aux, headers={'User-agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0'})
        c = r.content
        soup = BeautifulSoup(c, "html.parser")
        lst_soups.append(soup)
    return lst_soups

0 个答案:

没有答案