我已经设置了所有标题信息，但是仍然无法获取数据

Question

我已经使用浏览器获取标题信息，并设置了所有标题信息，但仍然无法正常获取数据。

    import requests
url='https://www.lagou.com/jobs/positionAjax.jsoncity=%E4%B8%8A%E6%B5%B7&needAddtionalResult=false'
headers={
'Host':'www.lagou.com',
'Connection':'keep-alive',
'Content-Length':'22',
'Pragma':'no-cache',
'Cache-Control':'no-cache',
'Origin':'https://www.lagou.com',
'X-Anit-Forge-Code':'0',
'User-Agent':'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',
'Content-Type':'application/x-www-form-urlencoded; charset=UTF-8',
'Accept':'application/json, text/javascript, */*; q=0.01',
'X-Requested-With':'XMLHttpRequest',
'X-Anit-Forge-Token':'None',
'Referer':'https://www.lagou.com/jobs/list_php?labelWords=&fromSearch=true&suginput=',
'Accept-Encoding':'gzip,deflate,br',
'Accept-Language':'en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7'
}
data={
    'first':'true',
    'pn':1,
    'kd':'php'
}
resp=requests.post(url=url,data=data,headers=headers)
print(resp.text)

Answer 1

请求标头中的

cookie也是needed，用于伪装正常的Web浏览器操作：

您可以将以下关键字段保留在cookie参数中：

cookies = {
    'HMACCOUNT':'your_HMACCOUNT_value',
    'LGRID':'your_LGRID_value',
    'JSESSIONID':'your_JSESSIONID_value',
    'user_trace_token': 'your_user_trace_token_value',
    'PRE_LAND': 'your_PRE_LAND_value',
}

# all these value you can find in Browser Developer Mode, you know how to get it man.

然后在您的请求中添加cookie参数

resp=requests.post(url=url, data=data, cookies=cookies,  headers=headers)
print(resp.text)

这是总代码，希望对您有帮助：

import requests

url = 'https://www.lagou.com/jobs/positionAjax.json?px=new&city=%E5%8C%97%E4%BA%AC&needAddtionalResult=false'
headers = {
    'Accept': 'application/json, text/javascript, */*; q=0.01',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'zh-CN,zh;q=0.9',
    'Connection': 'keep-alive',
    'Content-Length': '22',
    'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
    'Host': 'www.lagou.com',
    'Origin': 'https://www.lagou.com',
    'Referer': 'https://www.lagou.com/jobs/list_PHP?px=new&city=%E5%8C%97%E4%BA%AC',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',
    'X-Anit-Forge-Code': '0',
    'X-Anit-Forge-Token': 'None',
    'X-Requested-With': 'XMLHttpRequest',

}

data = {
    'first': 'true',
    'pn': 1,
    'kd': 'php'
}

cookies = {
    'HMACCOUNT': '04549EEF16D48382',
    'LGRID': '20190111154031-2c79703b-1574-11e9-990e-525400f775ce',
    'JSESSIONID': 'ABAAABAABEEAAJAC6C939E47B1C56C2C11F0557901B4343',
    'user_trace_token': '20190111141900-c94dbdcb-1568-11e9-9819-525400f775ce',
    'PRE_LAND': 'https%3A%2F%2Fpassport.lagou.com%2Flogin%2Flogin.html%3Fsignature%3DD82D1A9585AE6ED5D3AEA34E5CD1C9B0%26service%3Dhttp%25253A%25252F%25252Fwww.lagou.com%25252Fjobs%25252F%26action%3Dlogin%26serviceId%3Dlagou%26ts%3D1547187540500;',
}

resp = requests.post(url=url, data=data, cookies=cookies, headers=headers)
print(resp.text)

Answer 2

来自http://docs.python-requests.org/en/master/user/quickstart/：

我认为问题在于您对目标URL（url ='https://www.lagou.com/jobs/positionAjax.jsoncity=%E4%B8%8A%E6%B5%B7&needAddtionalResult=false'）进行编码或“格式化”以获取信息的方式。 URL参数（变量）应通过使用键值对字典而不是没有正确格式化参数的绝对URL路径进行编码。例如：

payload = {'key1': 'value1', 'key2': ['value2', 'value3']}

r = requests.get('https://httpbin.org/get', params=payload)
print(r.url)
RESULT: https://httpbin.org/get?key1=value1&key2=value2&key2=value3

您通常希望在URL的查询字符串中发送某种数据。如果您是手动构建网址，则此数据会在网址中以问号（例如， httpbin.org/get?key=val。请求允许您使用params关键字参数将这些参数提供为字符串字典。例如，如果要将key1 = value1和key2 = value2传递给httpbin.org/get，则可以使用以下代码：

payload = {'key1': 'value1', 'key2': 'value2'}
r = requests.get('https://httpbin.org/get', params=payload)

您应该阅读以下内容的后续文章：http://docs.python-requests.org/en/master/user/quickstart/，以获取一些见识。