Question

我用scrapy制作网络蜘蛛并且出现了问题：我试图获取一组html数据。它包含我需要发送ajax请求的ID。但是，当我试图获取ajax数据和我用html获得的其他数据，它只是出错了。我怎么能解决它？这是我的代码：

class DoubanSpider(scrapy.Spider):

    name = "douban"
    allowed_domains = ["movie.douban.com"]
    start_urls = ["https://movie.douban.com/review/best"]

    def parse(self, response):
        for review in response.css(".review-item"):
            rev = Review()
            rev['reviewer'] = review.css("a[property='v:reviewer']::text").extract_first()
            rev['rating'] = review.css("span[property='v:rating']::attr(class)").extract_first()
            rev['title'] = review.css(".main-bd>h2>a::text").extract_first()
            number = review.css("::attr(id)").extract_first()
            f = scrapy.Request(url='https://movie.douban.com/j/review/%s/full' % number,
                                     callback=self.parse_full_passage)
            rev['comment'] = f
            yield rev

    def parse_full_passage(self, response):
        r = json.loads(response.body_as_unicode())
        html = r['html']
        yield html

Answer 1

我会试试这个：

 response = scrapy.Request(url='https://movie.douban.com/j/review/%s/full' % number)
 jsonresponse = json.loads(response.body_as_unicode())
 rev['comment'] = jsonresponse['html']

如果您需要，可能需要从html字段中提取内容。或者使用this url

Answer 2

您需要先完全解析HTML，然后将其作为meta传递给JSON的回调：

yield scrapy.Request(url='https://movie.douban.com/j/review/%s/full' % number,callback=self.parse_full_passage, meta={'rev': rev} )

接下来是JSON的回调：

def parse_full_passage(self, response):
    rev = response.meta["rev"]
    r = json.loads(response.body_as_unicode())
    .....
    yield rev

如何用scrapy处理ajax数据

2 个答案: