Question

我试图让公司＆＃39;从政府网站获取信息并使用Scrapy。我的蜘蛛代码如下。

蜘蛛代码

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider
from ..items import CompaniesHouseItem

class SpendolaterSpider(scrapy.Spider):
    name = 'spendolater'
    allowed_domains = ['beta.companieshouse.gov.uk']
    start_url = ['https://beta.companieshouse.gov.uk/company/10511127']

    custom_settings = {"DOWNLOAD_DELAY": 1,}

    def crawling(self, response):
        domain = "https://beta.companieshouse.gov.uk/company/"
        for url in response.css("a::attr('href')").extract():
            if not url.startswith('https://'):
                continue 
            if domain not in url:
                yield scrapy.Request(url, callback=self.parse)
            yield scrapy.Request(url, callback=self.parse_dir_contents)

    def parse_item(self, response):
        for contents in response.xpath('//*[@id="page-container"]'):
            item = CompaniesHouseItem()
            item["name"] = response.xpath('//*[@id="company-name"]').extract()
            item["location"] = response.xpath('//*[@id="content-container"]/dl/dd').extract()
            item['foundation'] = response.xpath('//*[@id="company-creation-date"]').extract()
            items['type'] = response.xpath('//*[@id="company-type"]').extract()
            items['SIC'] = response.xpath('//*[@id="sic0"]').extract()
            yield item

运行时不会显示任何错误，但不会提取任何信息。＆＃34;抓取0页（0页/分钟），刮0项（0项/分钟）＆＃34;运行后，命令行显示消息。

＆＃34; items.py＆＃34;文件如下

items.py

import scrapy

class CompaniesHouseItem(scrapy.Item):
    name = scrapy.Field()
    location = scrapy.Field()
    foundation = scrapy.Field()
    type = scrapy.Field()
    SIC = scrapy.Field()

输出如下。

输出

2018-03-14 17:51:56 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: companies_house)
2018-03-14 17:51:56 [scrapy.utils.log] INFO: Versions: lxml 4.1.1.0, libxml2 2.9.5, cssselect 1.0.3, parsel 1.4.0, w3lib 1.19.0, Twisted 17.9.0, Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:04:45) [MSC v.1900 32 bit (Intel)], pyOpenSSL 17.5.0 (OpenSSL 1.1.0g  2 Nov 2017), cryptography 2.1.4, Platform Windows-10-10.0.16299-SP0
2018-03-14 17:51:56 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'companies_house', 'DOWNLOAD_DELAY': 1, 'NEWSPIDER_MODULE': 'companies_house.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['companies_house.spiders']}
2018-03-14 17:51:57 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2018-03-14 17:51:57 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-03-14 17:51:57 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-03-14 17:51:57 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2018-03-14 17:51:57 [scrapy.core.engine] INFO: Spider opened
2018-03-14 17:51:57 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-03-14 17:51:57 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-03-14 17:51:57 [scrapy.core.engine] INFO: Closing spider (finished)
2018-03-14 17:51:57 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'finish_reason': 'finished',
 'finish_time': datetime.datetime(2018, 3, 14, 8, 51, 57, 239817),
 'log_count/DEBUG': 1,
 'log_count/INFO': 7,
 'start_time': datetime.datetime(2018, 3, 14, 8, 51, 57, 231826)}
2018-03-14 17:51:57 [scrapy.core.engine] INFO: Spider closed (finished)

任何建议都将受到高度赞赏。提前谢谢。

Answer 1

因为默认情况下，Scrapy会在start_urls（而不是start_url）中读取要抓取的第一个地址，然后开始使用parse方法解析（不是crawling）。尝试重命名操作并重新启动蜘蛛。

Answer 2

您没有def start_requets(self)但start_url因此Scrapy将从列表start_url中抓取网址并使用您的回调方法parse。

我的意思是，您遗失def parse(self, response)将def crawling(self, response)更改为def parse(self, response)

此外，您的代码逻辑完全错误，只需在编写代码之前考虑代码的流程。

将页面放在start_url，其中包含公司链接，列出页面我的意思是

然后创建def parse(self, response)并创建一个for循环来迭代每个公司链接。

Scrapy：爬行并刮掉0件物品

蜘蛛代码

items.py

输出

2 个答案: