尝试打开网址时,我在使用Mechanize时遇到406错误:
for url in urls:
if "http://" not in url:
url = "http://" + url
print url
try:
page = mech.open("%s" % url)
except urllib2.HTTPError, e:
print "there was an error opening the URL, logging it"
print e.code
logfile = open ("log/urlopenlog.txt", "a")
logfile.write(url + "," + "couldn't open this page" + "\n")
continue
else:
print "opening this URL..."
page = mech.open(url)
知道什么会导致406错误发生?如果我转到相关网址,我可以在浏览器中打开它。
答案 0 :(得分:2)
尝试根据浏览器发送的内容为您的请求添加标头;首先添加Accept
标题(406通常意味着服务器不喜欢您想要接受的内容)。
请参阅文档中的"Adding headers":
req = mechanize.Request(url)
req.add_header('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8')
page = mechanize.urlopen(req)
Accept
标头值取决于Chrome发送的标头。
答案 1 :(得分:0)
如果您想了解浏览器发送的标题,此网页会向您显示:https://www.whatismybrowser.com/detect/what-http-headers-is-my-browser-sending
接受'和'用户代理'标题应该足够了。这就是我为摆脱错误所做的工作:
#establish counter
j = 0
#Create headers for webpage
headers = {'User-Agent': 'Mozilla/5.0', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'}
#Create for loop to get through list of URLs
for url in URLs:
#Verify scraper agent so that web security systems don't block webpage scraping upon URL opening, with j as a counter
req = mechanize.Request(URLs[j], headers = headers)
#Open the url
page = mechanize.urlopen(req)
#increase counter
j += 1
你也可以尝试导入" urllib2"或" urllib"用于打开这些URL的库。语法是一样的。