Question

我正在构建一个简单的程序来浏览URL列表并使用美丽的汤来提取其内容。我只是试图遍历列表并检索html，但我一直收到以下错误：

Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "C:\ProgramData\Anaconda3\lib\http\client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\ProgramData\Anaconda3\lib\http\client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\ProgramData\Anaconda3\lib\http\client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\ProgramData\Anaconda3\lib\http\client.py", line 1026, in _send_output
    self.send(msg)
  File "C:\ProgramData\Anaconda3\lib\http\client.py", line 964, in send
    self.connect()
  File "C:\ProgramData\Anaconda3\lib\http\client.py", line 1400, in connect
    server_hostname=server_hostname)
  File "C:\ProgramData\Anaconda3\lib\ssl.py", line 401, in wrap_socket
    _context=self, _session=session)
  File "C:\ProgramData\Anaconda3\lib\ssl.py", line 808, in __init__
    self.do_handshake()
  File "C:\ProgramData\Anaconda3\lib\ssl.py", line 1061, in do_handshake
    self._sslobj.do_handshake()
  File "C:\ProgramData\Anaconda3\lib\ssl.py", line 683, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/thoma/PycharmProjects/fyp/urls_and_prep/parsing_html.py", line 17, in <module>
    response = urllib.request.urlopen(req)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 526, in open
    response = self._open(req, data)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 544, in _open
    '_open', req)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 504, in _call_chain
    result = func(*args)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 1320, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)>

我的程序很简单，但我无法理解或找到任何好的资源来确定究竟发生了什么/如何处理它。我知道它与SSL证书有关，但我不确定如何使用它们或在哪里安装它们等等。我只是在这个问题上有点不知所措。从来没有真正使用过SSL。非常感谢任何指导或帮助。这是代码：

import urllib.request
from bs4 import BeautifulSoup

file = open("all_urls.txt", "r")

for line in file:
    print(line)

    try:
        response = urllib.request.urlopen(line)
        html = response.read()
    except ValueError:
        print(ValueError)
        continue
    soup = BeautifulSoup(html, 'lxml')
    print(soup.get_text())

Answer 1

您使用的是Windows还是Linux？这个问题似乎不是在Python上，而是在Anaconda或OS中。您可以尝试一些简单的解决方案，例如：1 - 使用除Anaconda之外的其他Python安装来执行scrypt。 2 - 使用virtualenv隔离操作系统的组件。

Answer 2

以下将解决问题。但是请确保不要在生产中使用，因为它将在不验证SSL证书的情况下工作 -

import urllib
from bs4 import BeautifulSoup
import ssl

# This is a temporary fix .Be carefule of malicious links
context = ssl._create_unverified_context()
file = open("all_urls.txt", "r")

for line in file:
    print(line)

    try:
        response = urllib.request.urlopen(line, context=context)
        html = response.read()
    except ValueError:
        print(ValueError)
        continue
    soup = BeautifulSoup(html, 'lxml')
    print(soup.get_text())

SSL：Windows上的CERTIFICATE_VERIFY_FAILED错误

2 个答案: