Question

我正在尝试废弃网站，但我正在使用的start_url未被抓取。我在行print(response)中查看发生了什么以及我得到的输出是什么（似乎忽略了start_url以支持https://www.purplebricks.com/search?ref=header）：

c:\Users\andrew\Documents\Big Data Project\Data Collectors\PurpleBricks\Purplebricks>scrapy crawl purplebricks
2017-02-05 22:18:31 [scrapy] INFO: Scrapy 1.0.5 started (bot: Purplebricks)
2017-02-05 22:18:31 [scrapy] INFO: Optional features available: ssl, http11
2017-02-05 22:18:31 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'Purplebricks.spiders', 'SPIDER_MODULES': ['Purplebricks.spiders'], 'BOT_NAME': 'Purplebricks'}
2017-02-05 22:18:31 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState
2017-02-05 22:18:31 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2017-02-05 22:18:31 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2017-02-05 22:18:31 [scrapy] INFO: Enabled item pipelines:
2017-02-05 22:18:31 [scrapy] INFO: Spider opened
2017-02-05 22:18:31 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-02-05 22:18:31 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-02-05 22:18:32 [scrapy] DEBUG: Crawled (200) <GET https://www.purplebricks.com/search?ref=header#?q&location=se3&page=1&latitude=51.4688273310245&longitude=0.0176656312203414&searchType=ForSale&sortBy=1> (referer: None)
<200 https://www.purplebricks.com/search?ref=header>
2017-02-05 22:18:32 [scrapy] INFO: Closing spider (finished)
2017-02-05 22:18:32 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 235,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 13413,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2017, 2, 5, 22, 18, 32, 326000),
 'log_count/DEBUG': 2,
 'log_count/INFO': 7,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2017, 2, 5, 22, 18, 31, 913000)}
2017-02-05 22:18:32 [scrapy] INFO: Spider closed (finished)

我正在使用的代码如下。虽然我需要强制驱动程序拉起网页，但我可以尝试使用selenium - 即使使用selenium也无法打开start_url

import scrapy

from Purplebricks.items import PurplebricksItem

class purplebricksSpider(scrapy.Spider):
    name = "purplebricks"
    allowed_domains = ["purplebricks.com"]
    start_urls = ["https://www.purplebricks.com/search?ref=header#?q&location=se3&page=1&latitude=51.4688273310245&longitude=0.0176656312203414&searchType=ForSale&sortBy=1",
                  ]

    def parse(self, response):
        print(response)
        for sel in response.xpath('//*[@class="row properties"]'):
            item = PurplebricksItem()
            prices = sel.xpath('//*/p/span[@data-bind="formatCurrency: property.marketPrice, roundDecimalsTo: 0"]/text()').extract()
            prices = [price.strip() for price in prices]
            property_ids = sel.xpath('//*[@class="title"]/div[@class="primary"]/a/@href').re(r'(?<=property-for-sale\/)(.*?)(?=\/#)')
            property_ids = [property_id.strip() for property_id in property_ids]
            addresses = sel.xpath('//*[@class="title"]/div[@class="primary"]/a/p[@class="type"]/text()').extract()
            addresses = [address.strip() for address in addresses]
            descriptions = sel.xpath('//*[@class="title"]/div[@class="primary"]/a/p[@class="address"]/text()').extract()
            descriptions = [description.strip() for description in descriptions]

            result = zip(prices, property_ids, addresses, descriptions)
            for price, property_id, address, description in result:
                item['price'] = price
                item['property_id'] = property_id
                item['description'] = description
                item['address'] = address
                yield item

关于这里发生了什么的任何观点？

Answer 1

从网址中删除#。它有特殊的意义。通常情况下，右侧有字符，它描述了当前页面上的位置，但scrapy不会检查var string = "Lorem ipsum dolor sit amet ..."; var chars = {}; string.split("").forEach(function(chr) { if (chars.hasOwnProperty(chr)) { chars[chr]++; } else { chars[chr] = 1; } }); console.log(Object.keys(chars)); console.log(chars);的严格大小，并删除所有内容。

python scrapy忽略start_url

1 个答案: