Question

使用wget

下载网站时出现错误

代码：

import threading
import urllib.request
import os
import re
import time
import json

def wget(url):
                #self.url = url 
    data = os.popen('wget -qO- %s'% url).read()
    return data

print (wget("http://jamesholm.se/dj.php"))

错误：

Traceback (most recent call last):
  File "stand-alone-check-url.py", line 13, in <module>
    print (wget("http://jamesholm.se/dj.php"))
  File "stand-alone-check-url.py", line 10, in wget
    data = os.popen('wget -qO- %s'% url).read()
  File "/usr/local/lib/python3.4/codecs.py", line 313, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9a in position 13133: invalid start byte

如何克服此错误？

Answer 1

您不能将任意字节序列解码为utf-8编码文本：

>>> b'\xa9'.decode('utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa9 in position 0: invalid start byte

该页面表明它使用utf-8但服务器发送的实际数据不是utf-8。它发生了。

bs4.UnicodeDammit允许您处理编码不一致的数据：

import bs4 # $ pip install beautifulsoup4

print(bs4.UnicodeDammit.detwingle(b'S\x9aben  - Ostwind Rec').decode('utf-8'))
# -> Sšben  - Ostwind Rec

Answer 2

使用requests python模块。

而不是wget

>>> import requests
>>> data = requests.get("http://jamesholm.se/dj.php").text
>>> print(data)

一些'utf-8'编解码器无法解码字节

2 个答案: