尝试将this question的答案修改为我的问题但不成功。
以下是一些示例html代码：
<div id="provider-region-addresses">
<h3>Contact details</h3>
<h2 class="toggler nohide">Auckland</h2>
<dl class="clear">
<dt>More information</dt>
<dd>North Shore Hospital</dd><dt>Physical address</dt>
<dd>124 Shakespeare Rd, Takapuna, Auckland 0620</dd><dt>Postal address</dt>
<dd>Private Bag 93503, Takapuna, Auckland 0740</dd><dt>Postcode</dt>
<dd>0740</dd><dt>District/town</dt>
<dd>
North Shore, Takapuna</dd><dt>Region</dt>
<dd>Auckland</dd><dt>Phone</dt>
<dd>(09) 486 8996</dd><dt>Fax</dt>
<dd>(09) 486 8342</dd><dt>Website</dt>
<dd><a target="_blank" href="http://www.healthpoint.co.nz/default,61031.sm">http://www.healthpoint.co.nz/default,61031...</a></dd>
</dl>
<h2 class="toggler nohide">Auckland</h2>
<dl class="clear">
<dt>Physical address</dt>
<dd>Helensville</dd><dt>Postal address</dt>
<dd>PO Box 13, Helensville 0840</dd><dt>Postcode</dt>
<dd>0840</dd><dt>District/town</dt>
<dd>
Rodney, Helensville</dd><dt>Region</dt>
<dd>Auckland</dd><dt>Phone</dt>
<dd>(09) 420 9450</dd><dt>Fax</dt>
<dd>(09) 420 7050</dd><dt>Website</dt>
<dd><a target="_blank" href="http://www.healthpoint.co.nz/default,61031.sm">http://www.healthpoint.co.nz/default,61031...</a></dd>
</dl>
<h2 class="toggler nohide">Auckland</h2>
<dl class="clear">
<dt>Physical address</dt>
<dd>Warkworth</dd><dt>Postal address</dt>
<dd>PO Box 505, Warkworth 0941</dd><dt>Postcode</dt>
<dd>0941</dd><dt>District/town</dt>
<dd>
Rodney, Warkworth</dd><dt>Region</dt>
<dd>Auckland</dd><dt>Phone</dt>
<dd>(09) 422 2700</dd><dt>Fax</dt>
<dd>(09) 422 2709</dd><dt>Website</dt>
<dd><a target="_blank" href="http://www.healthpoint.co.nz/default,61031.sm">http://www.healthpoint.co.nz/default,61031...</a></dd>
</dl>
<h2 class="toggler nohide">Auckland</h2>
<dl class="clear">
<dt>More information</dt>
<dd>Waitakere Hospital</dd><dt>Physical address</dt>
<dd>55-75 Lincoln Rd, Henderson, Auckland 0610</dd><dt>Postal address</dt>
<dd>Private Bag 93115, Henderson, Auckland 0650</dd><dt>Postcode</dt>
<dd>0650</dd><dt>District/town</dt>
<dd>
Waitakere, Henderson</dd><dt>Region</dt>
<dd>Auckland</dd><dt>Phone</dt>
<dd>(09) 839 0000</dd><dt>Fax</dt>
<dd>(09) 837 6634</dd><dt>Website</dt>
<dd><a target="_blank" href="http://www.healthpoint.co.nz/default,61031.sm">http://www.healthpoint.co.nz/default,61031...</a></dd>
</dl>
<h2 class="toggler nohide">Auckland</h2>
<dl class="clear">
<dt>More information</dt>
<dd>Hibiscus Coast Community Health Centre</dd><dt>Physical address</dt>
<dd>136 Whangaparaoa Rd, Red Beach 0932</dd><dt>Postcode</dt>
<dd>0932</dd><dt>District/town</dt>
<dd>
Rodney, Red Beach</dd><dt>Region</dt>
<dd>Auckland</dd><dt>Phone</dt>
<dd>(09) 427 0300</dd><dt>Fax</dt>
<dd>(09) 427 0391</dd><dt>Website</dt>
<dd><a target="_blank" href="http://www.healthpoint.co.nz/default,61031.sm">http://www.healthpoint.co.nz/default,61031...</a></dd>
</dl>
</div>
这是我的蜘蛛;
from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from webhealth.items1 import WebhealthItem1
class WebhealthSpider(BaseSpider):
name = "webhealth_content1"
download_delay = 5
allowed_domains = ["webhealth.co.nz"]
start_urls = [
"http://auckland.webhealth.co.nz/provider/service/view/914136/"
]
def parse(self, response):
hxs = HtmlXPathSelector(response)
results = hxs.select('//*[@id="content"]/div[1]')
items1 = []
for result in results:
item = WebhealthItem1()
item['url'] = result.select('//dl/a/@href').extract()
item['practice'] = result.select('//h1/text()').extract()
item['hours'] = map(unicode.strip, result.select('//div/dl/dt[contains(text(),"Contact hours")]/following-sibling::dd[1]/text()').extract())
item['more_hours'] = map(unicode.strip, result.select('//div/dl/dt[contains(text(),"More information")]/following-sibling::dd[1]/text()').extract())
item['physical_address'] = map(unicode.strip, result.select('//div/dl/dt[contains(text(),"Physical address")]/following-sibling::dd[1]/text()').extract())
item['postal_address'] = map(unicode.strip, result.select('//div/dl/dt[contains(text(),"Postal address")]/following-sibling::dd[1]/text()').extract())
item['postcode'] = map(unicode.strip, result.select('//div/dl/dt[contains(text(),"Postcode")]/following-sibling::dd[1]/text()').extract())
item['district_town'] = map(unicode.strip, result.select('//div/dl/dt[contains(text(),"District/town")]/following-sibling::dd[1]/text()').extract())
item['region'] = map(unicode.strip, result.select('//div/dl/dt[contains(text(),"Region")]/following-sibling::dd[1]/text()').extract())
item['phone'] = map(unicode.strip, result.select('//div/dl/dt[contains(text(),"Phone")]/following-sibling::dd[1]/text()').extract())
item['website'] = map(unicode.strip, result.select('//div/dl/dt[contains(text(),"Website")]/following-sibling::dd[1]/a/@href').extract())
item['email'] = map(unicode.strip, result.select('//div/dl/dt[contains(text(),"Email")]/following-sibling::dd[1]/a/text()').extract())
items1.append(item)
return items1
从这里开始，如何将列表项解析到单独的行中，并在名称字段中使用相应的//h1/text()值？目前，我在一个单元格中获取每个Xpath项目的列表。是否与我声明Xpaths的方式有关？
由于
答案 0 :(得分：4)
首先，您使用的是results = hxs.select('//*[@id="content"]/div[1]')
results = hxs.select('//*[@id="content"]/div[1]')
for result in results:
...
将仅停留在div 一个div的第一个孩子<div id="content" class="clear">
您需要循环使用此<dl class="clear">...</dl>中的每个//*[@id="content"]/div[1]（使用//*[@id="content"]/div[@class="content"]进行维护可能会更容易）
results = hxs.select('//*[@id="content"]/div[@class="content"]/div/dl')
其次，在每次循环迭代中，您使用的是绝对XPath表达式（//div...）
result.select('//div/dl/dt[contains(text(), "...")]/following-sibling::dd[1]/text()')
这将选择所有 dd跟随dt匹配从文档根节点开始的文本内容。
查看this section in Scrapy docs了解详情。
您需要使用相对XPath表达式 - 在代表每个result的每个dl范围内相对，例如dt[contains(text(),"Contact hours")]/following-sibling::dd[1]/text()或./dt[contains(text(), "Contact hours")]/following-sibling::dd[1]/text()，
然而，“practice”字段仍然可以使用绝对XPath表达式//h1/text()，但您也可以设置一次变量practice，并在每个WebhealthItem1()实例中使用它
...
practice = hxs.select('//h1/text()').extract()
for result in results:
item = WebhealthItem1()
...
item['practice'] = practice
以下是您的蜘蛛在这些变化中的样子：
from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from webhealth.items1 import WebhealthItem1
class WebhealthSpider(BaseSpider):
name = "webhealth_content1"
download_delay = 5
allowed_domains = ["webhealth.co.nz"]
start_urls = [
"http://auckland.webhealth.co.nz/provider/service/view/914136/"
]
def parse(self, response):
hxs = HtmlXPathSelector(response)
practice = hxs.select('//h1/text()').extract()
items1 = []
results = hxs.select('//*[@id="content"]/div[@class="content"]/div/dl')
for result in results:
item = WebhealthItem1()
#item['url'] = result.select('//dl/a/@href').extract()
item['practice'] = practice
item['hours'] = map(unicode.strip,
result.select('dt[contains(.," Contact hours")]/following-sibling::dd[1]/text()').extract())
item['more_hours'] = map(unicode.strip,
result.select('dt[contains(., "More information")]/following-sibling::dd[1]/text()').extract())
item['physical_address'] = map(unicode.strip,
result.select('dt[contains(., "Physical address")]/following-sibling::dd[1]/text()').extract())
item['postal_address'] = map(unicode.strip,
result.select('dt[contains(., "Postal address")]/following-sibling::dd[1]/text()').extract())
item['postcode'] = map(unicode.strip,
result.select('dt[contains(., "Postcode")]/following-sibling::dd[1]/text()').extract())
item['district_town'] = map(unicode.strip,
result.select('dt[contains(., "District/town")]/following-sibling::dd[1]/text()').extract())
item['region'] = map(unicode.strip,
result.select('dt[contains(., "Region")]/following-sibling::dd[1]/text()').extract())
item['phone'] = map(unicode.strip,
result.select('dt[contains(., "Phone")]/following-sibling::dd[1]/text()').extract())
item['website'] = map(unicode.strip,
result.select('dt[contains(., "Website")]/following-sibling::dd[1]/a/@href').extract())
item['email'] = map(unicode.strip,
result.select('dt[contains(., "Email")]/following-sibling::dd[1]/a/text()').extract())
items1.append(item)
return items1
我还使用此代码创建了一个Cloud9 IDE项目。您可以在https://c9.io/redapple/so_19309960
播放