我正在尝试从POST请求中获取Cookie。以前,我使用urllib2,它仍然工作正常,但我想切换到更清晰的库python请求。不幸的是我在页面上收到错误。
由于请求是HTTPS,我无法嗅探它们以找出差异。
urllib2代码:
NINTENDO_LOGIN_PAGE = "https://id.nintendo.net/oauth/authorize/"
MIIVERSE_CALLBACK_URL = "https://miiverse.nintendo.net/auth/callback"
parameters = {'client_id': 'ead88d8d450f40ada5682060a8885ec0',
'response_type': 'code',
'redirect_uri': MIIVERSE_CALLBACK_URL,
'username': MIIVERSE_USERNAME,
'password': miiverse_password}
data = urlencode(parameters)
self.logger.debug(data)
req = urllib2.Request(NINTENDO_LOGIN_PAGE, data)
page = urllib2.urlopen(req).read()
self.logger.debug(page)
结果(好):
[...]
<div id="main-body">
<div id="try-miiverse">
<p class="try-miiverse-catch">A glimpse at some of the posts that are currently popular on Miiverse.</p>
<h2 class="headline">Miiverse Sampler</h2>
<div id="slide-post-container" class="list post-list">
[...]
请求代码:
req = requests.post(NINTENDO_LOGIN_PAGE, data=parameters)
self.logger.debug(req.text)
结果(不好):
[...]
<div id="main-body">
<h2 class="headline">Activity Feed</h2>
<div class="activity-feed content-loading-window">
<div>
<img src="https://d13ph7xrk1ee39.cloudfront.net/img/loading-image-green.gif" alt=""></img>
<p class="tleft"><span>Loading activity feed...</span></p>
</div>
</div>
<div class="activity-feed content-load-error-window none"><div>
<p>The activity feed could not be loaded. Check your Internet connection, wait a moment and then try reloading.</p>
<div class="buttons-content"><a href="/" class="button">Reload</a></div>
</div>
</div>
[...]
提前感谢您解决此问题的任何提示。
更新1:感谢大家的回复!
根据@abarnert的建议,我检查了重定向。
resp = urllib2.urlopen(req)
print(resp.geturl()) # https://miiverse.nintendo.net/
req = requests.post(NINTENDO_LOGIN_PAGE, data=parameters)
print(req.url) # https://miiverse.nintendo.net/
print(req.history) # (<Response [303]>, <Response [302]>)
看起来他们都遵循重定向,但最终都在同一个地方。
的urllib2:
{
"args": {},
"data": "",
"files": {},
"form": {
"client_id": "ead88d8d450f40ada5682060a8885ec0",
"response_type": "code",
"redirect_uri": "https://miiverse.nintendo.net/auth/callback",
"username": "Wiwiweb",
"password": "password"
},
"headers": {
"Accept-Encoding": "identity",
"Connection": "close",
"Content-Length": "170",
"Content-Type": "application/x-www-form-urlencoded",
"Host": "httpbin.org",
"User-Agent": "Python-urllib/2.7"
},
"json": null,
"origin": "24.85.129.188",
"url": "http://httpbin.org/post"
}
请求:
{
"args": {},
"data": "",
"files": {},
"form": {
"client_id": "ead88d8d450f40ada5682060a8885ec0",
"response_type": "code",
"redirect_uri": "https://miiverse.nintendo.net/auth/callback",
"username": "Wiwiweb",
"password": "password"
},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate, compress",
"Connection": "close",
"Content-Length": "170",
"Content-Type": "application/x-www-form-urlencoded"
"Host": "httpbin.org",
"User-Agent": "python-requests/1.2.3 CPython/2.7.5 Windows/7"
},
"json": null,
"origin": "24.85.129.188"
"url": "http://httpbin.org/post",
}
看起来有些标题略有不同。我没有任何其他想法,所以我不妨尝试完全复制urllib2标头。欺骗用户代理可能就是这样。
更新2:我已将这些标头添加到“请求”请求中:
headers = {'User-Agent': 'Python-urllib/2.7',
'Accept-Encoding': 'identity'}
我仍然得到相同的结果......现在请求之间的唯一区别是“请求”有一个额外的标题:"Accept": "*/*"
。我不确定这是不是问题。
它可能来自重定向吗?
答案 0 :(得分:0)
好吧,我没有完全解决“为什么”重定向不同,但我发现了使用请求从哪里获取cookie。
我认为两个库之间的区别与它们处理重定向的方式有关。所以我检查了两个请求的历史记录。对于'请求'来说就像执行req.history
一样简单,但对于urllib2,我使用了这段代码:
class MyHTTPRedirectHandler(urllib2.HTTPRedirectHandler):
def http_error_302(self, req, fp, code, msg, headers):
print("New request:")
print(headers)
return urllib2.HTTPRedirectHandler.http_error_302(self, req, fp, code, msg, headers)
opener = urllib2.build_opener(MyHTTPRedirectHandler, urllib2.HTTPCookieProcessor())
urllib2.install_opener(opener)
检查历史记录让我看到'requests'请求在第一次重定向期间有'set-cookie'标题(所以第二个请求中有三个),但最后没有。这对我来说已经足够了,因为我知道现在该去哪里:req.history[1].cookies['ms']
作为一个奇怪的说明,由于我添加到urllib2请求的那一点,它开始返回与'requests'请求相同的东西!甚至将其改为:
class MyHTTPRedirectHandler(urllib2.HTTPRedirectHandler):
pass
opener = urllib2.build_opener(MyHTTPRedirectHandler, urllib2.HTTPCookieProcessor())
urllib2.install_opener(opener)
足以使它完全改变对同样返回的“请求”的响应(我在问题中将其标记为“坏”)。
我很难过,但知道在哪里找到这个饼干对我来说已经足够了。也许有些好奇和无聊的人会有兴趣找到原因。
谢谢大家的帮助:)