Question

我是scath的新手，并且正在努力加入与错误的绝对和相对链接：请求URL中缺少方案。奇怪的是，当我打印URL时，它似乎是正确的URL。

我已尝试过stackoverflow的许多不同解决方案，似乎没有取得任何进展，任何帮助都将不胜感激！

我的代码：

import scrapy

class CHSpider(scrapy.Spider):
    name = "ch_companydata"
    allowed_domains = ['download.companieshouse.gov.uk']
    start_urls = ['http://download.companieshouse.gov.uk/en_output.html']

    custom_settings = {
        'DOWNLOAD_WARNSIZE': 0
    }

    def parse(self, response):
        relative_url = response.xpath("//div[@class='grid_7 push_1 omega']/ul[2]/li[1]/a/@href").extract()[0]
        download_url = response.urljoin(relative_url)
        print(download_url)
        yield { 
            'file_urls': download_url
        }

错误讯息：

2017-08-01 09:46:36 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: companieshouse)
2017-08-01 09:46:36 [scrapy.utils.log] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'companieshouse.spiders', 'SPIDER_MODULES': ['companieshouse.spiders'], 'ROBOTSTXT_OBEY': True, 'BOT_NAME': 'companieshouse'}
2017-08-01 09:46:36 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats']
2017-08-01 09:46:37 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-08-01 09:46:37 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-08-01 09:46:37 [scrapy.middleware] INFO: Enabled item pipelines:
['scrapy.pipelines.files.FilesPipeline']
2017-08-01 09:46:37 [scrapy.core.engine] INFO: Spider opened
2017-08-01 09:46:37 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-08-01 09:46:37 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-08-01 09:46:37 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://download.companieshouse.gov.uk/robots.txt> (referer: None)
2017-08-01 09:46:37 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://download.companieshouse.gov.uk/en_output.html> (referer: None)
http://download.companieshouse.gov.uk/BasicCompanyData-2017-08-01-part1_5.zip
2017-08-01 09:46:37 [scrapy.core.scraper] ERROR: Error processing {'file_urls': u'http://download.companieshouse.gov.uk/BasicCompanyData-2017-08-01-part1_5.zip'}
Traceback (most recent call last):
  File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "c:\python27\lib\site-packages\scrapy\pipelines\media.py", line 79, in process_item
    requests = arg_to_iter(self.get_media_requests(item, info))
  File "c:\python27\lib\site-packages\scrapy\pipelines\files.py", line 382, in get_media_requests
    return [Request(x) for x in item.get(self.files_urls_field, [])]
  File "c:\python27\lib\site-packages\scrapy\http\request\__init__.py", line 25, in __init__
    self._set_url(url)
  File "c:\python27\lib\site-packages\scrapy\http\request\__init__.py", line 58, in _set_url
    raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url: h
2017-08-01 09:46:37 [scrapy.core.engine] INFO: Closing spider (finished)
2017-08-01 09:46:37 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 480,
 'downloader/request_count': 2,
 'downloader/request_method_count/GET': 2,
 'downloader/response_bytes': 8455,
 'downloader/response_count': 2,
 'downloader/response_status_count/200': 1,
 'downloader/response_status_count/404': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2017, 8, 1, 8, 46, 37, 415000),
 'log_count/DEBUG': 3,
 'log_count/ERROR': 1,
 'log_count/INFO': 7,
 'response_received_count': 2,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2017, 8, 1, 8, 46, 37, 69000)}
2017-08-01 09:46:37 [scrapy.core.engine] INFO: Spider closed (finished)

Answer 1

file_urls字段需要包含网址列表所以你应该产生这种内容：

    yield { 
        'file_urls': [download_url]
    }

Scrapy：结合Absolute＆amp;相对链接 - 缺少架构

1 个答案: