Question

[在此处输入图片描述] [1]作为标题，我无法使用python requests获取网站的内容，但是可以使用postman

我曾尝试将网站与邮递员联系起来并获取网页内容，但是当我执行如下所示的邮递员生成的代码时，却无法实现，相反，我在python 2和3中收到了错误500状态代码。

导入请求

url = "https://www.screener.reuters.wallst.com/Stock/US/ResultsTable"

querystring = {"returnChoice":"","page":"2","sortBy":"RCCMultexCompanyName","sortDir":"A","quickscreen":"gaarp","criteria":"%5B%22StockUniverse%22%2C%22PriceEarnings%22%2C%22EPSGrowthRate%22%2C%22Region%22%2C%22SectorAndIndustry%22%2C%22PEGRatio%22%5D","Currency":"USD","PEGRatio":"%7B%22view%22%3A%22button%22%2C%22button_inputs%22%3A%5B%5D%2C%22range_inputs%22%3A%22LSS%7C1%22%7D","SectorAndIndustry":"%7B%22industries%22%3A%5B%2257111%22%2C%2257112%22%2C%2257121%22%2C%2257131%22%2C%2257132%22%2C%2257211%22%2C%2257212%22%5D%7D","Region":"%7B%22countries%22%3A%5B%22TW%22%5D%7D","EPSGrowthRate":"%7B%22view%22%3A%22button%22%2C%22button_inputs%22%3A%5B%5D%2C%22range_inputs%22%3A%22GTR%7C15%22%7D","PriceEarnings":"%7B%22view%22%3A%22button%22%2C%22button_inputs%22%3A%5B%5D%2C%22range_inputs%22%3A%22GEQ%7C0%7CLEQ%7C15%22%7D","StockUniverse":"%7B%22button_inputs%22%3A%5B%22LIKE%7CUnited%2BStates%22%2C%22NOTLIKE%7CUnited%2BStates%22%5D%7D","OriginalCurrency":"USD%0A"}

headers = {
    'cache-control': "no-cache",
    }

response = requests.request("GET", url, headers=headers, params=querystring)

print(response.text)

我除了得到200个状态码外，实际上得到了500个。奇怪的是，即使我已经填写了标头，邮递员也可以得到正确的结果而python无法得到正确的结果。

headers = {
    'Accept-Encoding': "gzip, deflate, br",
    'Accept-Language': "zh-TW,zh;q=0.9,en-US;q=0.8,en;q=0.7",
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.90 Safari/537.36',
    'Content-Type': "application/x-www-form-urlencoded",
    'Accept': "application/json",
    'X-Requested-With': "XMLHttpRequest",
    'Connection': "keep-alive",
    'cache-control': "no-cache",
    }



The below url can get content if you connect it with browser or postman but  can't  get if you connect it with python reqeusts or urllib. 

https://www.screener.reuters.wallst.com/Stock/US/ResultsTable?returnChoice=&page=2&sortBy=RCCMultexCompanyName&sortDir=A&quickscreen=gaarp&criteria=%5B%22StockUniverse%22%2C%22PriceEarnings%22%2C%22EPSGrowthRate%22%2C%22Region%22%2C%22SectorAndIndustry%22%2C%22PEGRatio%22%5D&Currency=USD&PEGRatio=%7B%22view%22%3A%22button%22%2C%22button_inputs%22%3A%5B%5D%2C%22range_inputs%22%3A%22LSS%7C1%22%7D&SectorAndIndustry=%7B%22industries%22%3A%5B%2257111%22%2C%2257112%22%2C%2257121%22%2C%2257131%22%2C%2257132%22%2C%2257211%22%2C%2257212%22%5D%7D&Region=%7B%22countries%22%3A%5B%22TW%22%5D%7D&EPSGrowthRate=%7B%22view%22%3A%22button%22%2C%22button_inputs%22%3A%5B%5D%2C%22range_inputs%22%3A%22GTR%7C15%22%7D&PriceEarnings=%7B%22view%22%3A%22button%22%2C%22button_inputs%22%3A%5B%5D%2C%22range_inputs%22%3A%22GEQ%7C0%7CLEQ%7C15%22%7D&StockUniverse=%7B%22button_inputs%22%3A%5B%22LIKE%7CUnited%2BStates%22%2C%22NOTLIKE%7CUnited%2BStates%22%5D%7D&OriginalCurrency=USD

And below is postman screenshot
  [1]: https://i.stack.imgur.com/DO8ev.png

Answer 1

我发现邮递员会秘密存储cookie，即使在新桌面邮递员版本生成的邮递员的python代码中没有cookie时也是如此。

获取正确内容的方法是请求在URL下发布带有相关参数的帖子 https://www.screener.reuters.wallst.com/Stock/US/Results 它将出现500错误，但是通过此会话，我可以获取问题的内容。

邮递员获得状态代码200，不带标题GET METHOD，但python请求获得500错误

1 个答案: