Question

我一直在尝试使用urllib将https网站的内容提取到python中。我使用了4行代码。

import urllib
fhand = urllib.urlopen('https://www.tax.service.gov.uk/view-my-valuation/list-valuations-by-postcode?postcode=w1a&startPage=1#search-results')

for line in fhand:
    print line.strip()

当从python打开页面时，连接似乎正在工作。但是，我在输出中的标题，标题和段落标题中收到了一些不同的错误消息，如下所示。我原本期望输出是一系列html标签，其中包含网站上可用的数据，例如地址，基本费率和案例编号（即如果我进入谷歌浏览器开发人员的元素，可用的html）。任何人都可以指导我将这些数据转换成python吗？

谢谢＆amp;此致

<!DOCTYPE html>
<html class="no-branding"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Your request cannot be completed - GOV.UK</title>
<link href="/edge-assets/gone.css" media="screen" rel="stylesheet" type="text/css">
<!--[if lte IE 8]><link href="/edge-assets/ie.css" media="screen" rel="stylesheet" type="text/css"><![endif]-->
<link rel="icon" href="/edge-assets/govukfavicon.ico" type="image/x-icon" />
</head>
<body>
<div id="wrapper">
<div id="banner" role="banner">
<div class="inner">
<h1>
<a href="https://www.gov.uk/">
<img src="/edge-assets/govuk-logo.png" alt="GOV.UK">
</a>
</h1>
</div>
</div>
<div id="message" role="main">
<div class="inner">
<div id="detail">
<h2>Sorry, there was a problem handling your request.</h2>
<p class="call-to-action">Please try again shortly.</p>
</div>
<div id="footer">
</div>
</div>
</div>
</div>
</body></html>

Answer 1

当未指定用户代理或不希望用户代理时，某些网站会阻止请求。因此，请尝试在请求的标题中添加用户代理

import urllib2


headers = {'User-Agent': 'Mozilla/5.0'}
url = 'https://www.tax.service.gov.uk/view-my-valuation/list-valuations-by-postcode?postcode=w1a&startPage=1#search-results'
req = urllib2.Request(url, headers=HEADERS)
f = urllib2.urlopen(req)
s = f.read()
print s
f.close()

或者您可以pip install requests并使用print(requests.get(url).text)

使用urllib将数据从https站点提取到python中（您的请求无法完成错误）

1 个答案: