Question

试图从Google首页信息面板中检索餐馆地址，但收到“ urllib.error.HTTPError：HTTP错误403：禁止” 错误和程序未运行。我比较熟悉python网络抓取功能，请帮忙。

    import urllib.request, urllib.parse, urllib.error
    from bs4 import BeautifulSoup
    import ssl
    import json
    import re
    import sys
    import warnings

    if not sys.warnoptions:
        warnings.simplefilter("ignore")

   #get google URL.
    url = "https://www.google.com/search?q=barbeque%20nation%20-%20noida"
    request = urllib.request.Request(url)
    response = urllib.request.urlopen(request)

    page = fromstring(response)

    soup = BeautifulSoup(page, 'url.parser')

    the_page = soup.prettify("utf-8")
    hotel_json = {}

    for line in soup.find_all('script',attrs={"type" : 
    "application/ld+json"}):
        details = line.text.strip()
        details = json.loads(details)

        hotel_json["address"]["LrzXr"]=details["address"]["streetAddress"]

        break
    with open(hotel_json["name"]+".html", "wb") as file:
        file.write(html)

    with open(hotel_json["name"]+".json", 'w') as outfile:
        json.dump(hotel_json, outfile, indent=4)

Answer 1

添加用户代理标题

request = urllib.request.Request(url, headers = {'User-Agent' : 'Mozilla/5.0'})

尝试在任何不幸的地方或餐厅抓取Google首页地址

1 个答案: