Question

我试着去凑这个网站，但我的蜘蛛返回空的JSON，我不能设法了解问题的所在。我的代码和/或方法有问题吗？谢谢

import scrapy
import json

class SrealitySpider(scrapy.Spider):
    name = 'sreality'
    allowed_domains = ['www.sreality.cz/en']


    def start_requests(self):
        yield scrapy.Request(url="https://www.sreality.cz/api/en/v2/estates?category_main_cb=1&category_type_cb=2&locality_region_id=10&per_page=20&tms=1548939428469",
                            callback=self.parse_id)

    def parse_id(self, response):
        data = json.loads(response.body)
        estates = data.get("_embedded").get("estates")
        for estate in estates:
            yield scrapy.Request(url="https://www.sreality.cz/api{0}?tms=1548942301694".format(estate.get("_links").get("self").get("href")), callback=self.parse)


    def parse(self, response):
        estate = json.loads(response.body)

        yield {
                 "lat": estate.get("map").get("lat"),
                 "lon": estate.get("map").get("lon"),
                 "title": estate.get("name").get("value").replace(u'\xa0', u' '),
                 "address": estate.get("locality").get("value"),
                 "Price (czk)": estate.get("price_czk").get("value").replace(u'\xa0', u' '),
                 "nearby (m)": {prox.get("name"): prox.get("distance") for prox in estate.get("poi")},
                 "attributes": {attrib.get("name"): attrib.get("value") for attrib in estate.get("items")}

                }

Answer 1

修复您的allowed_domains：allowed_domains = ['www.sreality.cz'] 因为蜘蛛得到

2019-02-01 15:32:50 [scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to 'www.sreality.cz': <GET https://www.sreality.cz/api/en/v2/estates/1034202716?region_tip=220858?tms=1548942301694>

匆匆返回空的json文件

1 个答案: