我正在使用BeautifulSoup解析有关汽车生产的数据(另请参阅我的first question):
from bs4 import BeautifulSoup
import string
html = """
<h4>Production Capacity (year)</h4>
<div class="profile-area">
Vehicle 1,140,000 units /year
</div>
<h4>Output</h4>
<div class="profile-area">
Vehicle 809,000 units ( 2016 )
</div>
<div class="profile-area">
Vehicle 815,000 units ( 2015 )
</div>
<div class="profile-area">
Vehicle 836,000 units ( 2014 )
</div>
<div class="profile-area">
Vehicle 807,000 units ( 2013 )
</div>
<div class="profile-area">
Vehicle 760,000 units ( 2012 )
</div>
<div class="profile-area">
Vehicle 805,000 units ( 2011 )
</div>
"""
soup = BeautifulSoup(html, 'lxml')
for item in soup.select("div.profile-area"):
produkz = item.text.strip()
produkz = produkz.replace('\n',':')
prev_h4 = str(item.find_previous_sibling('h4'))
if "Models" in prev_h4:
models=produkz
else:
models=""
if "Capacity" in prev_h4:
capacity=produkz
else:
capacity=""
if "( 2015 )" in produkz:
prod15=produkz
else:
prod15=""
if "( 2016 )" in produkz:
prod16=produkz
else:
prod16=""
if "( 2017 )" in produkz:
prod17=produkz
else:
prod17=""
print(models+';'+capacity+';'+prod15+';'+prod16+';'+prod17)
我的问题是,所有匹配的HTML事件(“div.profile-area”)的下一个循环都会覆盖我的结果:
;Vehicle 1,140,000 units /year;;;;;;
;;;;;;Vehicle 809,000 units ( 2016 );
;;;;;Vehicle 815,000 units ( 2015 );;
;;;;Vehicle 836,000 units ( 2014 );;;
;;;Vehicle 807,000 units ( 2013 );;;;
;;Vehicle 760,000 units ( 2012 );;;;;
;;;;;;;
我想要的结果是:
;Vehicle 1,140,000 units /year;Vehicle 760,000 units ( 2012 );Vehicle 807,000 units ( 2013 );Vehicle 836,000 units ( 2014 );Vehicle 815,000 units ( 2015 );Vehicle 809,000 units ( 2016 );
如果你能给我一个更好的方法来构建我的代码,我会很高兴的。提前谢谢。
答案 0 :(得分:1)
这是我的解决方案,您需要处理每个元素标记并相应地解析它。我进一步解决了您的问题,并提供了一种更灵活的方式来访问每个数据值。希望它有所帮助。
import re
from bs4 import BeautifulSoup
html_doc = """
<h4>Production Capacity (year)</h4>
<div class="profile-area">
Vehicle 1,140,000 units /year
</div>
<h4>Output</h4>
<div class="profile-area">
Vehicle 809,000 units ( 2016 )
</div>
<div class="profile-area">
Vehicle 815,000 units ( 2015 )
</div>
<div class="profile-area">
Vehicle 836,000 units ( 2014 )
</div>
<div class="profile-area">
Vehicle 807,000 units ( 2013 )
</div>
<div class="profile-area">
Vehicle 760,000 units ( 2012 )
</div>
<div class="profile-area">
Vehicle 805,000 units ( 2011 )
</div>"""
soup = BeautifulSoup(html_doc, 'html.parser')
h4_elements = soup.find_all('h4')
profile_areas = soup.find_all('div', attrs={'class': 'profile-area'})
print('\n')
print("++++++++++++++++++++++++++++++++++++")
print("Element counts")
print("++++++++++++++++++++++++++++++++++++")
print("Total H4: {}".format(len(h4_elements)))
print("++++++++++++++++++++++++++++++++++++")
print("Total profile-area: {}".format(len(profile_areas)))
print("++++++++++++++++++++++++++++++++++++")
print('\n')
for i in h4_elements:
print("++++++++++++++++++++++++++++++++++++")
print(i.text.rstrip().lstrip())
print("++++++++++++++++++++++++++++++++++++")
del profile_areas[0]
for j in profile_areas:
raw = re.sub('[^A-Za-z0-9]+', ' ', j.text.replace(',','').lstrip().rstrip())
raw = raw.rstrip()
el = raw.split(' ')
print('Type: {} '.format(el[0]))
print('Sold: {} {} '.format(el[1], el[2]))
print('Year: {} '.format(el[3]))
print("++++++++++++++++++++++++++++++++++++")
输出如下:
++++++++++++++++++++++++++++++++++++
Production Capacity (year)
++++++++++++++++++++++++++++++++++++
Type:Vehicle
Sold: 809000 units
Year: 2016
++++++++++++++++++++++++++++++++++++
Type:Vehicle
Sold: 815000 units
Year: 2015
++++++++++++++++++++++++++++++++++++
Type:Vehicle
Sold: 836000 units
Year: 2014
++++++++++++++++++++++++++++++++++++
Type:Vehicle
Sold: 807000 units
Year: 2013
++++++++++++++++++++++++++++++++++++
Type:Vehicle
Sold: 760000 units
Year: 2012
++++++++++++++++++++++++++++++++++++
Type:Vehicle
Sold: 805000 units
Year: 2011
++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++
Output
++++++++++++++++++++++++++++++++++++
Type:Vehicle
Sold: 815000 units
Year: 2015
++++++++++++++++++++++++++++++++++++
Type:Vehicle
Sold: 836000 units
Year: 2014
++++++++++++++++++++++++++++++++++++
Type:Vehicle
Sold: 807000 units
Year: 2013
++++++++++++++++++++++++++++++++++++
Type:Vehicle
Sold: 760000 units
Year: 2012
++++++++++++++++++++++++++++++++++++
Type:Vehicle
Sold: 805000 units
Year: 2011
++++++++++++++++++++++++++++++++++++
答案 1 :(得分:0)
我建议你将每个条目存储在字典中,然后你可以在最后轻松提取你想要的字段(你似乎不想要2011年?):
NA
这会显示:
from bs4 import BeautifulSoup
import re
html = """
<h4>Production Capacity (year)</h4>
<div class="profile-area">
Vehicle 1,140,000 units /year
</div>
<h4>Output</h4>
<div class="profile-area">
Vehicle 809,000 units ( 2016 )
</div>
<div class="profile-area">
Vehicle 815,000 units ( 2015 )
</div>
<div class="profile-area">
Vehicle 836,000 units ( 2014 )
</div>
<div class="profile-area">
Vehicle 807,000 units ( 2013 )
</div>
<div class="profile-area">
Vehicle 760,000 units ( 2012 )
</div>
<div class="profile-area">
Vehicle 805,000 units ( 2011 )
</div>
"""
soup = BeautifulSoup(html, 'lxml')
units = {}
for item in soup.find_all(['h4', 'div']):
if item.name == 'h4':
for h4 in ['capacity', 'output', 'models']:
if h4 in item.text.lower():
break
elif item.get('class', [''])[0] == 'profile-area':
vehicle = item.get_text(strip=True)
if h4 == 'output':
re_year = re.search(r'\( (\d+) \)', vehicle)
if re_year:
year = re_year.group(1)
else:
year = 'unknown'
units[year] = vehicle
else:
units[h4] = vehicle
req_fields = ['models', 'capacity', '2012', '2013', '2014', '2015', '2016']
print(';'.join([units.get(field, '') for field in req_fields]))
正则表达式用于从车辆输入中提取年份。然后将其用作字典中的键。
对于pastebin中的HTML,它给出了:
;Vehicle 1,140,000 units /year;Vehicle 760,000 units ( 2012 );Vehicle 807,000 units ( 2013 );Vehicle 836,000 units ( 2014 );Vehicle 815,000 units ( 2015 );Vehicle 809,000 units ( 2016 )