Question

cityURL='https://en.wikipedia.org/wiki/Elko,_Nevada'

def createObj(url):
    html = urlopen(url)
    bsObj = BeautifulSoup(html, 'lxml')
    return bsObj

bsObj1 = createObj(cityURL)

table1 = bsObj1.find("table", {"class":"infobox geography vcard"})
incorporated = table1.find("th", text='Incorporated (city)').findNext('td').get_text()

table1.find("th", text='. Total') # Problem here, due to the special dot, I cannot identify the "th"

我希望得到以下数据：

，共计17.6 土地，17.6 水，0.0

Answer 1

页面上的“。”不是“点”。它是一个unicode字符 BULLET （\ u2022）。

您可以使用python的regex（re）模块来实现此目的。

更新后的代码如下所示：

import re
cityURL='https://en.wikipedia.org/wiki/Elko,_Nevada'

def createObj(url):
    html = urlopen(url)
    bsObj = BeautifulSoup(html, 'lxml')
    return bsObj

bsObj1 = createObj(cityURL)

table1 = bsObj1.find("table", {"class":"infobox geography vcard"})
incorporated = table1.find("th", text='Incorporated (city)').findNext('td').get_text()

pattern = re.compile(r'Total')
table1.find("th", text=pattern)

或者，您可以使用比beautifulsoup快很多的lxml模块。

import requests
from lxml import html

cityURL='https://en.wikipedia.org/wiki/Elko,_Nevada'
r = requests.get(cityURL)
root = html.fromstring(r.content)

def normalize(text) : 
    return ''.join([i if ord(i) < 128 else ' ' for i in text]).strip().split()[0]

val_list = [(normalize(root.xpath('//table[@class="infobox geography vcard"]//tr[./th/text()="Area"]/following-sibling::tr[{}]//text()'.format(str(val)))[1]), normalize(root.xpath('//table[@class="infobox geography vcard"]//tr[./th/text()="Area"]/following-sibling::tr[{}]//text()'.format(str(val)))[3])) for val in xrange(1,4)]
print(val_list)

上面的代码将输出：

[(u'Total', u'17.6'), (u'Land', u'17.6'), (u'Water', u'0.0')]

Python Web Scraping特殊字符提取数据

1 个答案: