Question

我是在命令提示符下完成的：

scrapy shell "http://192.168.99.100:8050/render.html?url="https://merojob.com/category/banking-insurance-financial-services/""

，并想抓取网址部分，所以我做到了，它奏效了：

response.xpath('//*[@id = "search_bar"]/@data-path').extract()

它返回了这个：

['/category/banking-insurance-financial-services/']

现在，我将其复制到名为scrap.py的项目脚本中。它包含以下代码。

import scrapy
from scrapy_splash import SplashRequest

class MerojobSpider(scrapy.Spider):
    name = 'merojob'

    def start_request(self):
        yield SplashRequest(
        url = 'https://merojob.com/category/banking-insurance-financial-services/',
        callback = self.parse,
        )
    def parse(self, response):
        #Job = response.xpath('//*[@class="col-md-10 col-sm-10 job-search-desc-blk"]/h2/a/text()').extract()
        datapath =  response.xpath('//*[@id = "search_bar"]/@data-path').extract()

        for item in zip(datapath):
            scraped_info = {
                'datapath':item[0],

            }
            yield scraped_info

但是不会输出任何结果。

2019-07-07 14:33:14 [scrapy.utils.log] INFO: Scrapy 1.6.0 started (bot: MnoogleData)
2019-07-07 14:33:14 [scrapy.utils.log] INFO: Versions: lxml 4.3.4.0, libxml2 2.9.5, cssselect 1.0.3, parsel 1.5.1, w3lib 1.20.0, Twisted 19.2.1, Python 3.6.2 (v3.6.2:5fd33b5, Jul  8 2017, 04:57:36) [MSC v.1900 64 bit (AMD64)], pyOpenSSL 19.0.0 (OpenSSL 1.1.1c  28 May 2019), cryptography 2.7, Platform Windows-10-10.0.17134-SP0
2019-07-07 14:33:14 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'MnoogleData', 'DOWNLOAD_DELAY': 3, 'DUPEFILTER_CLASS': 'scrapy_splash.SplashAwareDupeFilter', 'FEED_FORMAT': 'csv', 'FEED_URI': 'MnoogleData.csv', 'NEWSPIDER_MODULE': 'MnoogleData.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['MnoogleData.spiders']}
2019-07-07 14:33:14 [scrapy.extensions.telnet] INFO: Telnet Password: cbc0a96eb361d32a
2019-07-07 14:33:15 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats']
2019-07-07 14:33:15 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy_splash.SplashCookiesMiddleware',
 'scrapy_splash.SplashMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2019-07-07 14:33:15 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy_splash.SplashDeduplicateArgsMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2019-07-07 14:33:15 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2019-07-07 14:33:15 [scrapy.core.engine] INFO: Spider opened
2019-07-07 14:33:15 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-07-07 14:33:15 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2019-07-07 14:33:15 [scrapy.core.engine] INFO: Closing spider (finished)
2019-07-07 14:33:15 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'finish_reason': 'finished',
 'finish_time': datetime.datetime(2019, 7, 7, 8, 48, 15, 175967),
 'log_count/INFO': 9,
 'start_time': datetime.datetime(2019, 7, 7, 8, 48, 15, 159021)}
2019-07-07 14:33:15 [scrapy.core.engine] INFO: Spider closed (finished)

在这里我使用zip是因为我需要多个字段，但是为了简化起见，我仅使用一个字段，即datapath。我什至将所有必要的scrapy-splash设置都粘贴到settings.py文件中。我的代码有什么问题？

Answer 1

禁用 HttpProxy ， HttpAuth 和异地中间件以调试Scrapy shell。

尝试：

data.key

如果不起作用，请尝试以下命令：

scrapy shell http://192.168.99.100:8050/render.html?url="https://merojob.com/category/banking-insurance-financial-services/"

在scrapy shell上工作在scrapy项目中不起作用

1 个答案: