Scrapy-href链接内的简单文本抓取工具

时间:2018-10-24 17:12:00

标签: python python-3.x xpath scrapy

我刚被刮擦了两年,所以这可能是多余的问题。我正在尝试抓取一个简单的文本,该文本是href链接中的一个故事。我试过//a.​​text()使用刮板外壳,但没有刮板。我也想输出到数据库或文本文件(scrapy项目)。该站点是:http://www.nderf.org/Experiences/

编辑: 现在,我可以抓取链接的标题,但不能抓取@href标记中的文本(故事)。

import scrapy

from ..items import NdeItem


class NDESpider(scrapy.Spider):
    name='NDEstories'
    allowed_domains=["nderf.org"]
    headers=['author','desc','story']
    download_delay=1.5
    item = []

    start_urls = ["http://www.nderf.org/Archives/exceptional.html"]


    def start_requests(self):
        requests = []
        for item in self.start_urls:
            requests.append(scrapy.Request(url=item, headers={'Referer': 'https://www.google.com/'}))
            return requests


    def parse(self, response):
        #/ html / body / div[2] / section[2] / div / p / strong / span / a

         a_selectors=response.xpath('//a[@class="MiniLink"]')
         href_selector=response.xpath('@href')

         for selector in a_selectors:
            items = []
            item = NdeItem()  # type: NdeItem
            item['author'] = selector.xpath('text()').extract_first()
            item['desc'] = selector.xpath('@href').get()
            item['story'] = selector.xpath('@href/text()').get()
            items.append(item)

            # Create a new Request object
            request = response.follow(item['desc'], callback=self.parse)

            # Return it thanks to a generator

            with open ('log.txt','a') as f:
                f.write('author:{0}\n desc:{1}\n story:{2}\n'.format(item['author'],item['desc'], item['story']))
            yield request

1 个答案:

答案 0 :(得分:0)

response.xpath('.//a/@href').get()