Question

我正在尝试从Craigslist中提取一些信息并将其存储在JSON文件中，但信息存储有点错误。而不是拥有[标题，链接，位置，时间]的数组，我得到一个包含所有标题的数组，一个包含所有链接，等等。我的标题是错误的还是for循环本身错了？

from scrapy.spiders import BaseSpider
from scrapy.selector import HtmlXPathSelector
from scrapy.selector import Selector
from craigslist_sample.items import CraigslistSampleItem

class MySpider(BaseSpider):
 name = "craig"
 allowed_domains = ["craigslist.org"]
 start_urls = ["https://pittsburgh.craigslist.org/search/ccc"]

 def parse(self, response):
    titles = response.selector.xpath("//p[@class='row']")
    items = []
    for titles in titles:
        item = CraigslistSampleItem()
        item["title"] = titles.xpath("//span[@id='titletextonly']").extract()
        item["link"] = titles.xpath("a/@href").extract()
        item["location"] = titles.xpath("//small").extract()
        item["time"] = titles.xpath('//time').extract()
        items.append(item)
    return items

Answer 1

那是因为你的内部xpaths匹配从树根开始的元素。相反，您需要通过添加一个点来强制它们在每个项目的上下文中工作：

for title in titles:
    item = CraigslistSampleItem()
    item["title"] = title.xpath(".//span[@id='titletextonly']").extract()
    item["link"] = title.xpath("a/@href").extract()
    item["location"] = title.xpath(".//small").extract()
    item["time"] = title.xpath('.//time').extract()
    yield item

麻烦将scrapy导入json

1 个答案: