Question

我正在尝试从this网站上抓取文字。它返回这样的文本：

à¤¡à¤¾. à¤à¥à¤·à¤¬à¤¹à¤¾à¤¦à¥à¤° à¤¥à¤¾à¤ªà¤¾à¤²à¤¾à¤ à¤ªà¥à¤¤à¥à¤°à¥à¤¶à¥à¤, à¤à¤®à¥à¤°à¤¿à¤à¤¾à¤®à¤¾ à¤¤à¥à¤à¤¶à¥à¤°à¥à¤à¥ à¤¨à¤¿à¤§à¤¨

代替：

भारतीय विदेश सचिव गोखले आज नेपाल आउँदै.

当前代码：

headers = {
        'Connection': 'close',
        'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36',
    }

def get_url_soup(url):
        url_request = requests.get(url, headers=headers, allow_redirects=True)
        soup = BeautifulSoup(url_request.text, 'lxml')
        return soup

soup = get_url_soup('https://www.onlinekhabar.com/2019/03/753522')
title_card = soup.find('div', {'class': 'nws__title--card'})

Answer 1

使用EncodingDetector：

from bs4.dammit import EncodingDetector

headers = {
        'Connection': 'close',
        'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36',
    }

def get_url_soup(url):
        url_request = requests.get(url, headers=headers, allow_redirects=True)
        http_encoding = url_request.encoding if 'charset' in url_request.headers.get('content-type', '').lower() else None
        html_encoding = EncodingDetector.find_declared_encoding(url_request.content, is_html=True)
        encoding = html_encoding or http_encoding
        soup = BeautifulSoup(url_request.content, 'lxml', from_encoding=encoding)
        return soup

soup = get_url_soup('https://www.onlinekhabar.com/2019/03/753522')
title_card = soup.find('div', {'class': 'nws__title--card'})

print(title_card.text)

输出：

होमपेज / 
समाचार / 
राष्ट्रिय समाचार

भारतीय विदेश सचिव गोखले आज नेपाल आउँदै
प्रधानमन्त्रीलगायत शीर्ष नेतासँग भेट्ने 
.
.
.

使用Beautiful Soup抓取网站时出现编码错误

1 个答案: