Question

我目前正在使用BeautifulSoup。我似乎遇到了一些与编码有关的问题。

这是我的代码：

import requests
from bs4 import BeautifulSoup
req = requests.get('https://pythonprogramming.net/parsememcparseface/')
soup = BeautifulSoup(req.content.decode('utf-8','ignore'))
print(soup.find_all('p'))

这是我的错误：

 UnicodeEncodeError: 'ascii' codec can't encode character '\u1d90' in position 602: ordinal not in range(128)

任何帮助都将不胜感激。

Answer 1

请添加＆＃34; html5lib＆＃34;或＆＃34; html.parser＆＃34;

#!/usr/bin/python
# -*- coding: utf-8 -*-

...

# Python 3.6.0
soup = BeautifulSoup(req.content.decode('utf-8','ignore'), "html5lib")

# Python 2.7.12
soup = BeautifulSoup(req.content.decode('utf-8','ignore'), "html.parser")

Answer 2

我试图重现您在这里遇到的问题，但却无法重现。

这是我尝试过的。

shiny

Answer 3

我可以复制您的错误消息并消除麻烦的字符。

首先，此代码只是请求页面并尝试保存它。尝试失败并显示您看到的消息。我通过将页面转换为忽略丑陋字符代码然后将其转换回字符的字节来创建页面副本。现在可以成功保存页面。

我用它做汤并找到段落标签。

>>> from bs4 import BeautifulSoup
>>> import requests
>>> req = requests.get('https://pythonprogramming.net/parsememcparseface/').text
>>> open('c:/scratch/temp.htm', 'w').write(req)
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
  File "C:\Python34\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u1d90' in position 6702: character maps to <undefined>
>>> modReq = str(req.encode('utf-8', 'ignore'))
>>> open('c:/scratch/temp.htm', 'w').write(modReq)
12556
>>> soup = BeautifulSoup(modReq, 'lxml')
>>> paras = soup.findAll('p')
>>> len(paras)
12

Python 3美丽的汤网络刮痧

3 个答案: