406 Mechanize出错

时间:2012-12-22 21:46:32

标签: python mechanize

尝试打开网址时,我在使用Mechanize时遇到406错误:

for url in urls:
    if "http://" not in url: 
        url = "http://" + url
    print url
    try:
        page = mech.open("%s" % url)
    except urllib2.HTTPError, e:
        print "there was an error opening the URL, logging it"
        print e.code
        logfile = open ("log/urlopenlog.txt", "a")
        logfile.write(url + "," + "couldn't open this page" + "\n")
        continue
    else:
        print "opening this URL..."
        page = mech.open(url)

知道什么会导致406错误发生?如果我转到相关网址,我可以在浏览器中打开它。

2 个答案:

答案 0 :(得分:2)

尝试根据浏览器发送的内容为您的请求添加标头;首先添加Accept标题(406通常意味着服务器不喜欢您想要接受的内容)。

请参阅文档中的"Adding headers"

req = mechanize.Request(url)
req.add_header('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8')
page = mechanize.urlopen(req)

Accept标头值取决于Chrome发送的标头。

答案 1 :(得分:0)

如果您想了解浏览器发送的标题,此网页会向您显示:https://www.whatismybrowser.com/detect/what-http-headers-is-my-browser-sending

接受'和'用户代理'标题应该足够了。这就是我为摆脱错误所做的工作:

#establish counter
j = 0

#Create headers for webpage
headers = {'User-Agent': 'Mozilla/5.0', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'}

#Create for loop to get through list of URLs
for url in URLs:

    #Verify scraper agent so that web security systems don't block webpage scraping upon URL opening, with j as a counter
    req = mechanize.Request(URLs[j], headers = headers)

    #Open the url
    page = mechanize.urlopen(req)

    #increase counter
    j += 1

你也可以尝试导入" urllib2"或" urllib"用于打开这些URL的库。语法是一样的。