Question

我确定这是一个非常常见的问题，但我只是想了解一下发生了什么，这样我才能学会了。

问题：运行urllib.urlopen和requests.get都会将不同的HTML返回给浏览器中返回的内容。我已经禁用了Javascript，以防它是通过AJAX或其他东西，但问题仍然存在。我还检查了开发人员工具的XHR调用或其他方式，但没有找到任何内容。

Python代码：

headers = {'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Encoding':'gzip, deflate, sdch',
'Accept-Language':'en-GB,en-US;q=0.8,en;q=0.6',
'Cache-Control':'max-age=0',
'Connection':'keep-alive',
'Cookie':'tgaLiveAssistantVisible=false; ASP.NET_SessionId=c5wxbmboygytpn20gubpbh0o; .ASPXANONYMOUS=lLuVr11J1Tbq9zbQPk-ZpOFMQQ4gOlePIJpfGJwAnsFNV-1gUyvJpfb2ubnIPGZg8dQnlygyrKDj1KGf14vUehOx6Iwolslm0NZZkVbfdRvv9nBZhrmAiC-2MQijnBlmZPwinqBVVVrmBmVQavyOqUVzKWr8qapeBUO4nHWgzEy_1MqPx2njtCs0DvZ3IUdz0; __utmt=1; __utma=185625580.34928295.1483869478.1484065473.1484068673.4; __utmb=185625580.17.10.1484068673; __utmc=185625580; __utmz=185625580.1484065473.3.2.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided)',
'Host':'training.gov.au',
'Upgrade-Insecure-Requests':'1',
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'
}
# Load the RTO page
r = urllib.urlopen('https://training.gov.au/Organisation/Details/45001')
# r = requests.get('https://training.gov.au/Organisation/Details/45001', headers=headers)
soup = BeautifulSoup(r.read(),"html.parser")
print(soup)

返回HTML的底部：

<div class="display-row">
<div class="display-label">ABN:</div>
<div class="display-field-no-width">
</div>
</div>
</div>

而在浏览器中：

Request URL:http://training.gov.au/Organisation/Details/45001
Request Method:GET
Status Code:200 OK
Remote Address:117.53.170.212:80

返回HTML：

<div class="display-row">
<div class="display-label">ABN:</div>
<div class="display-field-no-width">
<a href="http://www.abr.business.gov.au/search.aspx?SearchText=90608470113"title="View organisation on ABN Lookup"rel="external">90 608 470 113
<span class="accessibilityOnly">(external link)</span>
<img src="/Content/images/openNewWindow.png" alt=""/>
</a>
</div>
</div>

有人可以向我解释为什么会发生这种情况吗？

Answer 1

使用请求：

import requests

url = 'https://training.gov.au/Organisation/Details/45001'
r = requests.get(url).text

出：

<div class="display-row">
                <div class="display-label">ABN:</div>
                <div class="display-field-no-width">
                <a href="http://www.abr.business.gov.au/search.aspx?SearchText=90608470113"title="View organisation on ABN Lookup"rel="external">90 608 470 113
<span class="accessibilityOnly">(external link)</span>
<img src="/Content/images/openNewWindow.png" alt=""/>
</a>

                </div>
            </div>

Answer 2

根据您的浏览器类型，

服务器可能会给您不同的响应。如果您希望得到与浏览器相同的响应，您可以查看selenium python绑定。

Python requests.get和urllib.urlopen将不同的HTML返回给浏览器

2 个答案: