Question

我正在尝试从每页15个广告格式的网页中抓取，然后单击下一页并获得接下来的15个广告。

由于某种原因，该脚本仅从一个页面抓取，而从不刮到另一页面。

这是我的脚本代码：

page_num = 10
curr_page = 1
i = 1
car_title, price_hrk, year_made, km_made, date_pub, temp = [], [], [], [], [], []
title = soup.find_all(class_="classified-title")
price_kn = soup.find_all(class_="price-kn")
info = soup.find_all(class_="info-wrapper")
date = soup.find_all("span", class_="date")


# while the current page is less then or equal to the page_num variable
while curr_page <= page_num:
    # make a request with a current page
    page = requests.get("https://www.oglasnik.hr/prodaja-automobila?page={}".format(curr_page))
    # pass it to beautiful soup
    soup = BeautifulSoup(page.content, "html.parser")

    # while i is less then 15 elements on the single site
    while i <= 15:
        # check for existance
        if title[i]:
            # append the value
            car_title.append(title[i].get_text())
        else:
            # append NaN
            car_title.append(np.nan)

        if price_kn[i]:
            price_hrk.append(price_kn[i].get_text())
        else:
            price_hrk.append(np.nan)

        if date[i]:
            date_pub.append(date[i].get_text())
        else:
            date_pub.append(np.nan)

        # dual values, so append both to a temporary list
        for tag in info[i].find_all("span", class_="classified-param-value"):
            for val in tag:
                temp.append(val)

        try:
            # if length of element is less then 5
            if len(temp[0]) < 5:
                # it's a year, append to year_made list
                year_made.append(temp[0])
            km_made.append(temp[2])
        except IndexError:
            # if index out of bound append NaN
            year_made.append(np.nan)
            km_made.append(np.nan)

        # reset temp
        temp = []
        # add 1 to i element
        i += 1

    # add 1 to current page
    curr_page += 1

现在，如果我打印出其中一个列表的长度，我将得到15。

有人可以告诉我我做错了什么或将我指向正确的方向吗？

谢谢。

Answer 1

您还需要重置i。就在

之前（或之后）

   curr_page += 1

添加：

   i = 1

它应该可以工作。

Python BeautifulSoup不抓取多个页面

1 个答案: