Question

我刚被刮擦了两年，所以这可能是多余的问题。我正在尝试抓取一个简单的文本，该文本是href链接中的一个故事。我试过//a.text（）使用刮板外壳，但没有刮板。我也想输出到数据库或文本文件（scrapy项目）。该站点是：http://www.nderf.org/Experiences/

编辑： 现在，我可以抓取链接的标题，但不能抓取@href标记中的文本（故事）。

import scrapy

from ..items import NdeItem


class NDESpider(scrapy.Spider):
    name='NDEstories'
    allowed_domains=["nderf.org"]
    headers=['author','desc','story']
    download_delay=1.5
    item = []

    start_urls = ["http://www.nderf.org/Archives/exceptional.html"]


    def start_requests(self):
        requests = []
        for item in self.start_urls:
            requests.append(scrapy.Request(url=item, headers={'Referer': 'https://www.google.com/'}))
            return requests


    def parse(self, response):
        #/ html / body / div[2] / section[2] / div / p / strong / span / a

         a_selectors=response.xpath('//a[@class="MiniLink"]')
         href_selector=response.xpath('@href')

         for selector in a_selectors:
            items = []
            item = NdeItem()  # type: NdeItem
            item['author'] = selector.xpath('text()').extract_first()
            item['desc'] = selector.xpath('@href').get()
            item['story'] = selector.xpath('@href/text()').get()
            items.append(item)

            # Create a new Request object
            request = response.follow(item['desc'], callback=self.parse)

            # Return it thanks to a generator

            with open ('log.txt','a') as f:
                f.write('author:{0}\n desc:{1}\n story:{2}\n'.format(item['author'],item['desc'], item['story']))
            yield request

Answer 1

response.xpath('.//a/@href').get()

Scrapy-href链接内的简单文本抓取工具

1 个答案: