用刮y来建造蜘蛛

时间:2018-07-03 20:26:58

标签: web-scraping scrapy web-crawler scrapy-spider

我正在构建一个简单的抓取工具,以从众筹网站获取数据。我运行刮板,然后尝试将数据输出到csv文件。不幸的是,我的csv文件一直空白。我尝试了许多建议,但似乎无济于事。这是我写的一个例子:

class Crowdfund_Helphopelive(scrapy.Spider):
name = "helphopelive_scraper"   

def start_requests(self):
    #make list of individual campaign sites
    start_urls = ["https://helphopelive.org/campaign/10798/",
                  "https://helphopelive.org/campaign/13083/"]
    for url in start_urls:
        yield scrapy.Request(url=url,callback=self.parse)

def parse(self, response):
    #parse individual campaign sites

    #create instance of the item
    item = HelphopeliveItem()

    #get amount raised
    item['amount_raised'] = response.xpath("//span[contains(@class,'teal')]/descendant::text()").extract()[0]
    #get the goal
    item['goal'] = response.xpath("//div[contains(@class,'profile-lockup__footer')]/p/text()").extract()[0]

    #get the url
    item['url'] = response.xpath("//meta[@property='og:url']/@content").extract()[0]

    yield item

将感谢您的帮助,以便我能理解自己的缺失。

0 个答案:

没有答案