Question

我使用beautifulsoup库从网页获取数据

http://open.dataforcities.org/details?4[]=2016

import urllib2
from bs4 import BeautifulSoup
soup = BeautifulSoup(urllib2.urlopen('http://open.dataforcities.org/details?4[]=2016').read())

现在soup如下所示（我只展示了它的一部分）：

soup('table):
[<table>\n<tr class="theme-cells" id="profile_indicators" ng-mouseover='updateIndicatorsScroll( "Profile Indicators" )'>\n<td class="theme-text">\n<h1>4 Profile Indicators</h1>\n</td>\n<td class="metrics">\n<div class="metric-p metric-title"></div>\n</td>\n</tr>\n<tr class="indicator-cells" ng-mouseover='updateIndicatorsScroll( "Profile Indicators" )' onmouseout="$(this).removeClass('indicator-cells-hover')" onmouseover="$(this).addClass('indicator-cells-hover')">\n<td class="indicator-text">\n<h2>4.1 Total city population (Profile)</h2>\n</td>\n<td class="metrics">\n<div class="metric-p metric-title"></div>\n<div class="metric-p-also bigger">669 469   (2015)</div>\n<div class="full-bar" style="width:100%">\n<div class="metric-bar" style="width:3.6411942141077174%; background-color:#ffffff"></div>\n</div>\n</td>\n</tr>\n<tr class="indicator-cells" ng-mouseover='updateIndicatorsScroll( "Profile Indicators" )' onmouseout="$(this).removeClass('indicator-cells-hover')" onmouseover="$(this).addClass('indicator-cells-hover')">\n<td class="indicator-text">\n<h2>4.2 City land area (Profile)</h2>\n</td>\n<td class="metrics">\n<div class="metric-p metric-title"></div>\n<div class="metric-p-also bigger">125 km\xb2 (2010)</div>\n<div class="full-bar" style="width:100%">\n<div class="metric-bar" style="width:1.9604120789229098%; background-color:#ffffff"></div>\n</div>\n</td>\n</tr>\n<tr class="indicator-cells" ng-mouseover='updateIndicatorsScroll( "Profile Indicators" )' onmouseout="$(this).removeClass('indicator-cells-hover')" onmouseover="$(this).addClass('indicator-cells-hover')">\n<td class="indicator-text">\n<h2>4.3 Population density (Profile)</h2>\n</td>\n<td class="metrics">\n<div class="metric-p metric-title"></div>\n<div class="metric-p-also bigger">5 354 /km\xb2 (2015)</div>\n<div class="full-bar" style="width:100%">\n<div class="metric-bar" style="width:27.890485963282238%; background-color:#ffffff"></div>\n</div>\n</td>\n</tr>\n<tr class="indicator-cells" ng-mouseover='updateIndicatorsScroll( "Profile Indicators" )'

如何从soup中提取数据？如果我按照Web scraping with Python中的示例进行操作，则会出现以下错误：

soup = BeautifulSoup(urllib2.urlopen('http://open.dataforcities.org/details?4[]=2016').read())

for row in soup('table', {'class': 'metrics'})[0].tbody('tr'):
    tds = row('td')
    print tds[0].string, tds[1].string

IndexError                                Traceback (most recent call last)
<ipython-input-71-d688ff354182> in <module>()
----> 1 for row in soup('table', {'class': 'metrics'})[0].tbody('tr'):
      2     tds = row('td')
      3     print tds[0].string, tds[1].string

IndexError: list index out of range

Answer 1

html中的表格没有“指标”类，因此您的表达式（'table.metrics'）会返回一个空列表，当您尝试选择第一个项目时，会为您提供IndexError。

由于页面上只有一个表，并且没有属性，因此您可以使用此表达式获取所有行：'table tr'

import urllib2
from bs4 import BeautifulSoup

html = urllib2.urlopen('http://open.dataforcities.org/details?4[]=2016').read()
soup = BeautifulSoup(html, 'html.parser')

for row in soup.select('table tr'):
    tds = row('td')
    print tds[0].text.strip(), tds[1].text.strip()

还要确保使用bs4而不是bs3，如果可能，请更新到Python3。

Answer 2

基本上这段代码会提取你的数据并将它们保存到csv中供你访问（顺便说一句，我觉得你的数据不完整）我建议打开该链接并将文件作为html文件下载，因为如果你尝试使用urlopener来提取它，则会出现UnicodeEncodeError

from bs4 import BeautifulSoup
import csv

soup=BeautifulSoup(open("Yourfile.html"),"html.parser")

f = csv.writer(open("file.csv", "w"))
f.writerow(["Information"]) 


h2s=soup.find_all("h2")

for h2 in h2s:
    name=h2.contents[0]
    f.writerow([name])

顺便说一下你想继续使用urlopener urllib2不再存在所以它实际上是

from urllib.request import urlopen
html =urlopen('http://open.dataforcities.org/details?4[]=2016').read()

Python：如何从文本中提取数据？

2 个答案: