Question

我试图用以下python代码从Yahoo的搜索结果中删除链接。我使用mechanize来实现浏览器实例和Beautiful soup来解析HTML代码。

问题是，这个脚本有时可以正常工作，有时会抛出错误：

WARNING:root:Some characters could not be decoded, and were replaced with REPLACEMENT CHARACTER.

很明显，我猜测它与编码和解码或gzip压缩相关的东西，但为什么有时工作，有时不工作？以及如何将它固定在工作中？

以下是代码。运行7-8次，你会注意到。

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import mechanize
import urllib
from bs4 import BeautifulSoup
import re

#mechanize emulates a Browser
br = mechanize.Browser()
br.set_handle_robots(False)
br.addheaders = [('User-agent','chrome')]

term = "stock market".replace(" ","+")
query = "https://search.yahoo.com/search?q=" + term

htmltext = br.open(query).read()
htm = str(htmltext)

soup = BeautifulSoup(htm)
#Since all results are located in the ol tag
search = soup.findAll('ol')

searchtext = str(search)

#Using BeautifulSoup to parse the HTML source
soup1 = BeautifulSoup(searchtext)
#Each search result is contained within div tag
list_items = soup1.findAll('div', attrs={'class':'res'})


#List of first search result
list_item = str(list_items)

for li in list_items:
    list_item = str(li)
    soup2 = BeautifulSoup(list_item)
    link = soup2.findAll('a')
    print link[0].get('href')
    print ""

这是输出屏幕截图： http://pokit.org/get/img/1d47e0d0dc08342cce89bc32ae6b8e3c.jpg

Answer 1

我在项目上编码时遇到了问题并开发了一个函数来获取我正在抓取的页面的编码 - 然后你可以解码为unicode以便你的函数尝试防止这些错误。 re：压缩你需要做的是开发你的代码，这样如果它遇到一个压缩文件就可以处理它。

from bs4 import BeautifulSoup, UnicodeDammit
import chardet
import re

def get_encoding(soup):
    """
    This is a method to find the encoding of a document.
    It takes in a Beautiful soup object and retrieves the values of that documents meta tags
    it checks for a meta charset first. If that exists it returns it as the encoding.
    If charset doesnt exist it checks for content-type and then content to try and find it.
    """
    encod = soup.meta.get('charset')
    if encod == None:
        encod = soup.meta.get('content-type')
        if encod == None:
            content = soup.meta.get('content')
            match = re.search('charset=(.*)', content)
            if match:
                encod = match.group(1)
            else:
                dic_of_possible_encodings = chardet.detect(unicode(soup))
                encod = dic_of_possible_encodings['encoding'] 
    return encod

处理压缩数据的链接http://www.diveintopython.net/http_web_services/gzip_compression.html

来自这个问题Check if GZIP file exists in Python

if any(os.path.isfile, ['bob.asc', 'bob.asc.gz']):
    print 'yay'

Python脚本有时会刮取html输出，有时不会

1 个答案: