我正在使用以下代码从网站上抓取数据。
# -*- coding: cp1252 -*-
import urllib2
import sys
from bs4 import BeautifulSoup
page = urllib2.urlopen('http://www.att.com/shop/wireless/plans-new.html#fbid=U-XD_DHOGEp').read()
soup = BeautifulSoup(page)
plans = soup.findAll('div', {"class": "planTitle"})
for plan in plans:
planname = u' '.join(plan.stripped_strings)
plantypes = soup.findAll('div', {"class":"top"})
prices = soup.findAll('div', {"class":"bottom"})
for plantype, price in zip(plantypes, prices):
plantype1 = u' '.join(plantype.stripped_strings)
price1 = u' '.join(price.stripped_strings)
print planname, plantype1, price1
问题：如果您浏览此代码中提到的网页，这些是4-5种类型的计划，每个计划有3个语音选项和2-3个数据选项。我想以这样的方式抓取数据：对于每个计划，我可以针对它获得各自的语音选项，然后是这些选项的月度价格。
我正在运行的代码现在返回计划名称+语音选项的所有可能组合。对于每个计划名称，我会得到大约20-30个条目，因为即使错误的计划名称+语音选项组合也会创建一个条目。例如。个人计划 - 550分钟 - 59.99美元，这个组合500分钟和59.99分是家庭计划的一部分。
我希望循环运行，以便只提取正确的Plan + Voice选项组合。
网页摘要： 对于每个计划，网页上都有一个框，其中包含与这些选项对应的语音选项和价格，我希望为每个框运行循环，但语音选项的元素+类组合及其价格不是唯一的。这就是为什么计划名称从其他boxex中获取价值的原因。
<div class="innerContainer">
<div class="planTitle">
<h2><a href="http://www.att.com/shop/wireless/plans/individualplans.html" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010">AT&T Individual Plans</a></h2>
</div>
<div class="planSubTitle">
<img src="/shopcms/media/att/2012/shop/wireless/promotions/Plans-CM_page/tiny-clock.jpg" alt="">
<p>Voice plan options:</p>
</div>
<!-- Begin three white boxes -->
<!-- Note, extra boxes can be added to the row with the following method -->
<!-- 1. Add more div containers inside .whiteBox -->
<!-- 2. Modify class names to boxes_one, boxes_two, boxes_three etc... (max six) -->
<div class="whiteBox">
<div class="boxes_three">
<a class="lnk-help tooltips fullBoxLink" href="#smartphone_individual_voice_450" onclick="window.location.href = 'http://www.att.com/shop/wireless/plans/voice/sku3830290.html?source=IC95ATPLP00PSP00L&wtExtndSource=spindvoice450';return false;" aria-describedby="smartphone_individual_voice_450" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010" title=""></a>
<span id="smartphone_individual_voice_450" class="tips" role="tooltip">$0.45/min. for additional minutes</span>
<div class="top">
<p class="stat">450</p>
<p class="statText">Minutes</p>
</div>
<div class="bottom">
<p>$39.99/mo.</p>
</div>
</div>
<div class="boxes_three">
<a class="lnk-help tooltips fullBoxLink" href="#smartphone_individual_voice_900" onclick="window.location.href = 'http://www.att.com/shop/wireless/plans/voice/sku3830292.html?source=IC95ATPLP00PSP00L&wtExtndSource=spindvoice900';return false;" aria-describedby="smartphone_individual_voice_900" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010" title=""></a>
<span id="smartphone_individual_voice_900" class="tips" role="tooltip">$0.40/min. for additional minutes</span>
<div class="top">
<p class="stat">900</p>
<p class="statText">Minutes</p>
</div>
<div class="bottom">
<p>$59.99/mo.</p>
</div>
</div>
<div class="boxes_three borderNone">
<a class="fullBoxLink" href="http://www.att.com/shop/wireless/plans/voice/sku3830293.html?source=IC95ATPLP00PSP00L&wtExtndSource=spindvoiceunlim" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010"></a>
<div class="top">
<p class="stat">Unlimited</p>
<p class="statText">Minutes</p>
</div>
<div class="bottom">
<p>$69.99/mo.</p>
</div>
</div>
</div>
<!-- End three white boxes -->
<!-- Begin left gray container -->
<div class="containerTwoThirds">
<div class="planSubTitle">
<img src="/shopcms/media/att/2012/shop/wireless/promotions/Plans-CM_page/tiny-globe.jpg" alt="">
<p>Data plan options:</p>
</div>
<div class="grayTwoThirds">
<div class="grayBox">
<a class="fullBoxLink" href="http://www.att.com/shop/wireless/services/dataplus300mb-smartphone4glte-sku5380269.html?source=IC95ATPLP00PSP00L&wtExtndSource=spinddata300mb" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010"></a>
<p class="stat"><strong>300MB</strong></p>
<p class="statText">$20.00/mo.</p>
</div>
<div class="grayBoxBreak"></div>
<div class="grayBox">
<a class="fullBoxLink" href="http://www.att.com/shop/wireless/services/datapro3gb-smartphone4glte-sku5470232.html?source=IC95ATPLP00PSP00L&wtExtndSource=spinddata3gb" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010"></a>
<p class="stat"><strong>3GB</strong></p>
<p class="statText">$30.00/mo.</p>
</div>
<div class="grayBoxBreak"></div>
<div class="grayBox">
<a class="fullBoxLink" href="http://www.att.com/shop/wireless/services/datapro5gb-smartphone4glte-sku5480228.html?source=IC95ATPLP00PSP00L&wtExtndSource=spinddata5gb" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010"></a>
<p class="stat"><strong>5GB</strong></p>
<p class="statText">$50.00/mo.</p>
</div>
</div>
</div>
<!-- End left gray container -->
<!-- Begin right gray container -->
<div class="containerThird">
<div class="planSubTitle">
<img src="/shopcms/media/att/2012/shop/wireless/promotions/Plans-CM_page/tiny-phone.jpg" alt="">
<p>Messaging plan options: <span class="fix"></span></p>
</div>
<div class="grayThird">
<div class="grayBox">
<a data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2012325" href="http://www.att.com/shop/wireless/services/messagingunlimited-sku1160055.html?source=IC95ATPLP00PSP00L&wtExtndSource=spindmessunlim" class="fullBoxLink"></a>
<p class="stat"><strong>ULTD</strong> MSGS</p>
<p class="statText">$20.00/mo.</p>
</div>
<div class="grayBoxBreak"></div>
<div class="grayBox last">
<p class="stat"><strong>PAY PER USE</strong></p>
<p class="statText">20¢/text <span class="lightGray">|</span> 30¢/pic/video</p>
</div>
</div>
</div>
<!-- End right gray container -->
<!-- Begin sub footer -->
<div class="bottomLinks">
<div class="links">
<a href="http://www.att.com/shop/wireless/plans/individualplans.html?taxoPlan=POSTPAID-INDIVIDUAL-CANADA&source=IC95ATPLP00PSP00L&wtExtndSource=spindcanada" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010">Nation with Canada Plans</a> | <a href="http://www.att.com/shop/wireless/plans/voice/sku5740279.html?source=IC95ATPLP00PSP00L&wtExtndSource=spindhomephone" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010">Unlimited Home Phone</a> | <a href="http://www.att.com/shop/wireless/plans/voice/sku3830294.html?source=IC95ATPLP00PSP00L&wtExtndSource=spindsenior" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010">Senior Plans</a>
</div>
<a class="shop_button" href="http://www.att.com/shop/wireless/devices/smartphones.html?source=IC95ATPLP00PSP00L&wtExtndSource=indshopsp" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010"><img src="/shopcms/media/att/2012/shop/wireless/promotions/Plans-CM_page/buttons/shop_smartphones.png" alt="Shop Smartphones" width="158" height="29"></a>
</div>
<!-- End sub footer -->
</div>
请帮助我解决这个问题，因为我是编程新手。
答案 0 :(得分：1)
从头开始重写这件事。没有评论，但它是非常自我解释的。字典中的lambda用于查找以某个字符串开头的属性。我引用了这个答案：https://stackoverflow.com/a/2830550/541208
我原以为你在findAll上使用soup，当你本应该使用plan.findAll时，但是它没有任何帮助，所以我只是重写了整个事情。
import urllib2
import sys
from bs4 import BeautifulSoup
page = urllib2.urlopen('http://www.att.com/shop/wireless/plans-new.html#fbid=U-XD_DHOGEp').read()
soup = BeautifulSoup(page)
#find the container for all the plans
tabcontent = soup.find('div', {"id": "smartphonePlans", "class": "tabcontent"})
containers = tabcontent.findAll('div', {"class": "innerContainer"})
for plan in containers:
planTitle = plan.find("div", {"class": "planTitle"})
if planTitle:
title = planTitle.find("a").text
print title
voiceBoxes = plan.find("div", {"class": "whiteBox"})
if voiceBoxes:
box3 = voiceBoxes.findAll("div", {"class": lambda x: x and x.startswith("boxes_")})
if box3:
for box in box3:
top = box.findAll("p")
minutes = u" ".join([tag.text for tag in top])
print "\t", minutes
哪个输出：
AT&T Individual Plans
450 Minutes $39.99/mo.
900 Minutes $59.99/mo.
Unlimited Minutes $69.99/mo.
AT&T Family Plans
550 Minutes $59.99/mo.
700 Minutes $69.99/mo.
1,400 Minutes $89.99/mo.
2,100 Minutes $109.99/mo.
Unlimited Minutes $119.99/mo.
AT&T Mobile Share Plans
1GB $40/mo. + $45/smartphone
4GB $70/mo. + $40/smartphone
6GB $90/mo. + $35/smartphone
10GB $120/mo.
15GB $160/mo. + $30/smartphone
20GB $200/mo.