Question

我正在尝试使所有链接从this website遍历下一页。我下面的脚本可以解析下一页的链接，直到10。但是，我不能越过该链接在该页面底部显示为10的链接。

我尝试过：

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

base = 'https://www.icab.es'
link = 'https://www.icab.es/?go=eaf9d1a0ec5f1dc58757ad6cffdacedb1a58854a600312cc82c494d2c55856f1e25c06b4b6fcee5ddabebfe2d30057589a86e9750b459e9d60598cc6e5c52a4697030b2b8921f29f'

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1; ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36'
    p = 1
    while True:
        r = s.get(link)
        soup = BeautifulSoup(r.text,"lxml")
        """some data I can fetch myself from current pages, so ignore this portion"""

        p+=1
        next_page = soup.select_one(f"a[title='{p}']")
        if next_page:
            link = urljoin(base,next_page.get("href"))
            print("next page:",link)
        else:
            break

如何从上面的网站获取所有下一页链接？

PS硒不是我想应付的选择。

Answer 1

您只需在href时获得">"的{{1}}

代码：

(p-1)%10 != 0

结果（import requests from bs4 import BeautifulSoup from urllib.parse import urljoin base = 'https://www.icab.es' link = 'https://www.icab.es/?go=eaf9d1a0ec5f1dc58757ad6cffdacedb1a58854a600312cc82c494d2c55856f1e25c06b4b6fcee5ddabebfe2d30057589a86e9750b459e9d60598cc6e5c52a4697030b2b8921f29f' with requests.Session() as s: s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1; ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36' p = 1 while True: r = s.get(link) soup = BeautifulSoup(r.text, "lxml") """some data I can fetch myself from current pages, so ignore this portion""" p += 1 if not ((p-1) % 10): next_page = soup.select_one(f"a[title='Següent']") else: next_page = soup.select_one(f"a[title='{p}']") if next_page: link = urljoin(base, next_page.get("href")) print("page", next_page.text, link)可被视为page >>）：

page ?1

Answer 2

我在使用SSL时遇到问题，因此我更改了此网站的默认ssl_context：

import ssl
import requests
import requests.adapters
from bs4 import BeautifulSoup


# adapted from https://stackoverflow.com/questions/42981429/ssl-failure-on-windows-using-python-requests/50215614
class SSLContextAdapter(requests.adapters.HTTPAdapter):
    def init_poolmanager(self, *args, **kwargs):
        ssl_context = ssl.create_default_context()
        # Sets up old and insecure TLSv1.
        ssl_context.options &= ~ssl.OP_NO_TLSv1_3 & ~ssl.OP_NO_TLSv1_2 & ~ssl.OP_NO_TLSv1_1
        ssl_context.minimum_version = ssl.TLSVersion.TLSv1
        kwargs['ssl_context'] = ssl_context
        return super(SSLContextAdapter, self).init_poolmanager(*args, **kwargs)


base = 'https://www.icab.es'
link = 'https://www.icab.es/?go=eaf9d1a0ec5f1dc58757ad6cffdacedb1a58854a600312cc82c494d2c55856f1e25c06b4b6fcee5ddabebfe2d30057589a86e9750b459e9d60598cc6e5c52a4697030b2b8921f29f'

with requests.session() as s:
    s.mount('https://www.icab.es', SSLContextAdapter())

    p = 1
    while True:
        print('Page {}..'.format(p))
        # r = urllib.request.urlopen(link, context=ssl_context)
        r = s.get(link)

        soup = BeautifulSoup(r.content, "lxml")

        for li in soup.select('li.principal'):
            print(li.get_text(strip=True))

        p += 1
        link = soup.select_one('a[title="{}"]'.format(p))
        if not link:
            link = soup.select_one('a[title="Següent"]')
            if not link:
                break
        link = base + link['href']

打印：

Page 1..
Sr./Sra. Martínez Gòmez, Marc
Sr./Sra. Eguinoa de San Roman, Roman
Sr./Sra. Morales Santiago, Maria Victoria
Sr./Sra. Bengoa Tortajada, Javier
Sr./Sra. Moralo Rodríguez, Xavier
Sr./Sra. Romagosa Huerta, Marta
Sr./Sra. Peña Moncho, Juan
Sr./Sra. Piñana Morera, Roman
Sr./Sra. Millán Sánchez, Antonio
Sr./Sra. Martínez Mira, Manel
Sr./Sra. Montserrat Rincón, Anna
Sr./Sra. Fernández Paricio, Maria Teresa
Sr./Sra. Ruiz Macián- Dagnino, Claudia
Sr./Sra. Barba Ausejo, Pablo
Sr./Sra. Bruna de Quixano, Jose Luis
Sr./Sra. Folch Estrada, Fernando
Sr./Sra. Gracia Castellón, Sonia
Sr./Sra. Sales Valls, Gemma Elena
Sr./Sra. Pastor Giménez-Salinas, Adolfo
Sr./Sra. Font Jané, Àlvar
Sr./Sra. García González, Susana
Sr./Sra. Garcia-Tornel Florensa, Xavier
Sr./Sra. Marín Granados, Alejandra
Sr./Sra. Albero Jové, José María
Sr./Sra. Galcerà Margalef, Montserrat
Page 2..
Sr./Sra. Chimenos Minguella, Sergi
Sr./Sra. Lacasta Casado, Ramón
Sr./Sra. Alcay Morandeira, Carlos
Sr./Sra. Ribó Massó, Ignacio
Sr./Sra. Fitó Baucells, Antoni
Sr./Sra. Paredes Batalla, Patricia
Sr./Sra. Prats Viñas, Francesc
Sr./Sra. Correig Ferré, Gerard
Sr./Sra. Subirana Freixas, Alba
Sr./Sra. Álvarez Crexells, Juan
Sr./Sra. Glaser Woloschin, Joan Nicolás
Sr./Sra. Nel-lo Padro, Francesc Xavier
Sr./Sra. Oliveras Dalmau, Rosa Maria
Sr./Sra. Badia Piqué, Montserrat
Sr./Sra. Fuentes-Lojo Rius, Alejandro
Sr./Sra. Argemí Delpuy, Marc
Sr./Sra. Espinoza Carrizosa, Pina
Sr./Sra. Ges Clot, Carla
Sr./Sra. Antón Tuneu, Beatriz
Sr./Sra. Schroder Vilalta, Andrea
Sr./Sra. Belibov, Mariana
Sr./Sra. Sole Lopez, Silvia
Sr./Sra. Reina Pardo, Luis
Sr./Sra. Cardenal Lagos, Manel Josep
Sr./Sra. Bru Galiana, David


...and so on.

无法使用请求解析网站的所有下一页链接

2 个答案: