Question

此web scraping with Scrapy有点过时的链接似乎Selector XPath已更改。

我复制时有

def parse(self, response):
    questions = Selector(response).xpath('//*[@id="question-header"]/h1/a')

但是链接上方的代码

class StackSpider(Spider):
    name = "stack"
    allowed_domains = ["stackoverflow.com"]
    start_urls = [
        "http://stackoverflow.com/questions?pagesize=50&sort=newest",
    ]

    def parse(self, response):
        questions = Selector(response).xpath('//div[@class="summary"]/h3')

        for question in questions:
            item = StackItem()
            item['title'] = question.xpath(
                'a[@class="question-hyperlink"]/text()').extract()[0]
            item['url'] = question.xpath(
                'a[@class="question-hyperlink"]/@href').extract()[0]
            yield item

如何使用新的选择器构造函数生成器？

这是

Spring data @transactional not rolling back with SQL Server and after runtimeexception

我们将以SO问题为例。

马修·丹尼尔斯的建议

In [4]: response                                                                                                                                                                                            
Out[4]: <200 https://stackoverflow.com/questions/27624141/spring-data-transactional-not-rolling-back-with-sql-server-and-after-runtimeexc>

In [5]: response.css(".question-hyperlink").xpath("@href").extract_first()                                                                                                                                  
Out[5]: '/questions/27624141/spring-data-transactional-not-rolling-back-with-sql-server-and-after-runtimeexc'

In [6]: response.css(".summary h3")                                                                                                                                                                         
Out[6]: []

In [7]: response.css("#question-header > h1 > a")                                                                                                                                                           
Out[7]: [<Selector xpath="descendant-or-self::*[@id = 'question-header']/h1/a" data='<a href="/questions/27624141/spring-data'>]

如何使用scrapy刮SO问题并存储在MongoDB中？

0 个答案: