Question

我正在尝试使用urllib从URL获得响应，然后使用bs4提取数据，但是在某些URL上我遇到了这种异常，并且

from urllib.request import urlopen

无法处理请求。出现以下错误：

已编辑：错误：（回溯完成）

Traceback (most recent call last):
  File "G:/Internships/Botnostic Solutions/AllScrapers/careerz360/test_bs4.py", line 74, in <module>
    html = urlopen(url_link)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 525, in open
    response = self._open(req, data)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 543, in _open
    '_open', req)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 503, in _call_chain
    result = func(*args)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 1360, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 1317, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 1240, in _send_request
    self.putrequest(method, url, **skips)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 1107, in putrequest
    self._output(request.encode('ascii'))
UnicodeEncodeError: 'ascii' codec can't encode character '\xe8' in position 18: ordinal not in range(128)

代码段：

from bs4 import BeautifulSoup
from urllib.request import urlopen
urls_list = []
with open('links_file.txt', 'r', encoding="utf-8") as linksFile:
    for l in linksFile:
        # url_parsed = urllib.parse.quote(l.strip())
        urls_list.append(l.strip())
linksFile.close()

将所有链接附加到列表，以便可以使用以下方法进一步处理它们： urlib.urlopen（）

这是在此处造成问题的链接：

urls_list = ['https://www.careerz360.com/pakistan/protègè-global-android-developer-karachi-jobs-108853']

重新设置代码：

for url in urls_list:
    html = urlopen(url)
    if html.getcode() == 200:
        response = BeautifulSoup(html.read(), "lxml")
        company_name = response.select('h2#business-name')[0].text.strip()

使用的不同编码：

我也尝试过使用不同的编码，例如： latin-1，utf-8和iso等，但是却遇到了相同的错误。如果可以的话，请给我建议解决方案。谢谢和问候

Answer 1

此代码对我有用。在file.txt中，我有一个URL，https://www.careerz360.com/pakistan/protègè-global-android-developer-karachi-jobs-108853：

import requests
from bs4 import BeautifulSoup

urls_list = []
with open('file.txt', 'r') as linksFile:
    for l in linksFile:
        urls_list.append(l.strip())

print(urls_list)

soup = BeautifulSoup(requests.get(urls_list[0]).text, 'lxml')

company_name = soup.select('h2#business-name')[0].text.strip()

print(company_name)

打印：

['https://www.careerz360.com/pakistan/protègè-global-android-developer-karachi-jobs-108853']
Protègè Global

我正在使用Python 3.6.8

编辑：使用urlopen，您需要quote() URL：

from urllib.request import urlopen
from urllib.parse import quote, urlparse
from bs4 import BeautifulSoup

urls_list = []
with open('file.txt', 'r') as linksFile:
    for l in linksFile:
        urls_list.append(l.strip())

print(urls_list)

p = urlparse(urls_list[0])
url = p.scheme + '://' + p.netloc + quote(p.path)

print(url)

html = urlopen(url)
soup = BeautifulSoup(html.read(), 'lxml')

company_name = soup.select('h2#business-name')[0].text.strip()

print(company_name)

打印：

['https://www.careerz360.com/pakistan/protègè-global-android-developer-karachi-jobs-108853']
https://www.careerz360.com/pakistan/prot%C3%A8g%C3%A8-global-android-developer-karachi-jobs-108853
Protègè Global

使用url通过urllib python请求时遇到编码问题

1 个答案: