Question

的网页

我想进入下一页，当我检查＆＃34; next＆＃34;我明白了：

<span class="pagerlink">
    <a href="#" id="next" title="Go to the next page">Next</a>
</span>

什么是正确的方法？

# -*- coding: utf-8 -*-
import scrapy
import json
from scrapy_splash import SplashRequest

class Manulife(scrapy.Spider):
    name = 'manulife'
    #allowed_domains = ['https://manulife.taleo.net/careersection/external_global/jobsearch.ftl?lang=en']
    start_urls = ['https://manulife.taleo.net/careersection/external_global/jobsearch.ftl?lang=en&location=1038']

    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(
            url,
            self.parse,
            args={'wait': 5},
            )   

    def parse(self, response):
        #yield {
        #   'demo' : response.css('div.absolute > span > a::text').extract()
        #     }
        urls = response.css('div.absolute > span > a::attr(href)').extract() 
        for url in urls:
            url = "https://manulife.taleo.net" + url
            yield SplashRequest(url = url, callback = self.parse_details, args={'wait': 5})
            #self.log("reaced22 : "+ url)

        #hitting next button
        #data = json.loads(response.text)
        #self.log("reached 22 : "+ data)
        #next_page_url = 

        if next_page_url:
           next_page_url = response.urljoin(next_page_url) 
           yield SplashRequest(url = next_page_url, callback = self.parse, args={'wait': 5})

    def parse_details(self,response):
        yield {
           'Job post' : response.css('div.contentlinepanel > span.titlepage::text').extract(),
           'Location' : response.xpath("//span[@id = 'requisitionDescriptionInterface.ID1679.row1']/text()").extract(),
           'Organization' : response.xpath("//span[@id = 'requisitionDescriptionInterface.ID1787.row1']/text()").extract(),
           'Date posted' : response.xpath("//span[@id = 'requisitionDescriptionInterface.reqPostingDate.row1']/text()").extract(),
           'Industry': response.xpath("//span[@id = 'requisitionDescriptionInterface.ID1951.row1']/text()").extract()
          }

如您所见，代码在点击下一页链接时包含SplashRequest。

我是抓手的新手，在某处我发现网站也可以像json一样返回响应。我试了一下，但它给了我一个错误＆＃34;没有json对象可以被解码＆＃34;

Answer 1

我认为像这样使用css选择器<!Doctype html> <html> <head> <title></title> <Style type="text/css"> #header{border:1px solid #000;} #1{border:1px solid #000; background-color:green;} </style> </head> <body> <script></script> <div id="header"> <h1>...</h1> </div><br> <div id="1"> <input type="text" name="JDate" placeholder="OnBoardDate"><br> <input type="text" name="RDate" placeholder="ReturnDate"><br> </div> </body> </html>可能会有效。

但是div将是最好的方法。你只需得到href属性

抓取包含锚标记<a href="#"> using scrapy

1 个答案: