Scrapy Spider在处理中出现错误

时间：2018-10-06 12:17:05

标签： python error-handling web-crawler scrapy-spider

我是第一次使用python编程并使用scrapy。我在网上抓取时遇到错误。我在此站点上使用了类似的帮助页面，甚至从头到尾都遵循了教程，但无济于事，我们将不胜感激。

错误说：蜘蛛错误处理http://quotes.toscrape.com/>（参考：无）追溯（最近一次通话）：

我发现了一些类似的代码，并且类似的代码运行良好 但是我写的不是

这是我的代码：

#-*-  coding: utf-8 -*-
import scrapy


class QuotesSpider(scrapy.Spider):
name = "quotes"
allowed_domains = ["quotes.toscrape.com"]
start_urls = (
    'http://quotes.toscrape.com/',
)

def parse(self, response):
    quotes = response.xpath('//*[@class="quote"]')
    for quote in quotes:
        text = quote.xpath('.//*[@class="text"]/text()').extract_first()
        author = quote.xpath('.//*[@itemprop="author"]/text()').extract_first()
        tags = quote.xpath('.//*[@itemprop="keywords"]/@content').extract_first()

        print '\n'
        print text
        print author
        print tags
        print '\n'

    next_page_url = response.xpath('//*[@class="next"]/a/@href').extract_first()
    absolute_next_page_url = response.urljoin(next_page_url)
    yield scrapy.Request(absolute_next_page_url)

这是命令提示符：

C:\Users\cancun\Desktop\Dosyalar\Coding\Phyton\Spider\quotes_spider>scrapy crawl quotes
2018-10-06 14:42:07 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: quotes_spider)
2018-10-06 14:42:07 [scrapy.utils.log] INFO: Versions: lxml 4.2.1.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.0, w3lib 1.19.0, Twisted 18.7.0, Python 2.7.15 |Anaconda, Inc.| (default, May  1 2018, 18:37:09) [MSC v.1500 64 bit (AMD64)], pyOpenSSL 18.0.0 (OpenSSL 1.0.2o  27 Mar 2018), cryptography 2.2.2, Platform Windows-10-10.0.16299
2018-10-06 14:42:07 [scrapy.crawler] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'quotes_spider.spiders', 'SPIDER_MODULES': ['quotes_spider.spiders'], 'BOT_NAME': 'quotes_spider'}
2018-10-06 14:42:07 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.logstats.LogStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.corestats.CoreStats']
2018-10-06 14:42:07 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-10-06 14:42:07 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-10-06 14:42:07 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2018-10-06 14:42:07 [scrapy.core.engine] INFO: Spider opened
2018-10-06 14:42:07 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-10-06 14:42:07 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6024
2018-10-06 14:42:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/> (referer: None)


2018-10-06 14:42:07 [scrapy.core.scraper] ERROR: Spider error processing <GET http://quotes.toscrape.com/> (referer: None)
Traceback (most recent call last):
File "c:\programdata\anaconda2\lib\site-packages\scrapy\utils\defer.py", line 102, in iter_errback
yield next(it)
File "c:\programdata\anaconda2\lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 30, in process_spider_output
for x in result:
File "c:\programdata\anaconda2\lib\site-packages\scrapy\spidermiddlewares\referer.py", line 339, in <genexpr>
return (_set_referer(r) for r in result or ())
File "c:\programdata\anaconda2\lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 37, in <genexpr>
return (r for r in result or () if _filter(r))
File "c:\programdata\anaconda2\lib\site-packages\scrapy\spidermiddlewares\depth.py", line 58, in <genexpr>
return (r for r in result or () if _filter(r))
File "C:\Users\cancun\Desktop\Dosyalar\Coding\Phyton\Spider\quotes_spider\quotes_spider\spiders\quotes.py", line 20, in parse
print text
File "c:\programdata\anaconda2\lib\encodings\cp437.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\u201c' in position 0: character maps to <undefined>
2018-10-06 14:42:07 [scrapy.core.engine] INFO: Closing spider (finished)
2018-10-06 14:42:07 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 218,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 2333,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2018, 10, 6, 11, 42, 7, 594000),
'log_count/DEBUG': 2,
'log_count/ERROR': 1,
'log_count/INFO': 7,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'spider_exceptions/UnicodeEncodeError': 1,
'start_time': datetime.datetime(2018, 10, 6, 11, 42, 7, 309000)}
2018-10-06 14:42:07 [scrapy.core.engine] INFO: Spider closed (finished)

谢谢！

2 个答案:

答案 0 :(得分：0)

问题是您的命令提示符使用cp437字符集，该字符集不支持您刚刚抓取的某些字符（例如，“ and”）

我认为不可能将这些字符打印到Windows Shell（也许我不知道，新的PowerShell支持它）。您可以创建一个新文件，为其选择一个丰富的字符集，然后将所有输出写入此处：

import io
out_file = io.open('output.txt', 'a', encoding='UTF-8')

如果您确实要使用操作系统使用的默认字符编码，请创建如下文件：

import io
out_file = io.open('output.txt', 'a', errors='backslashreplace')

使用后一种选项，您将看到类似的内容：

\u201cThe world as we have created it is a process of our thinking.\u201d

答案 1 :(得分：0)

我更改了一些代码，其工作方式与更新代码相同：

#-*-  coding: utf-8 -*-
import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    allowed_domains = ["quotes.toscrape.com"]
    start_urls = (
        'http://quotes.toscrape.com/',
    )

    def parse(self, response):
        quotes = response.xpath('//*[@class="quote"]')
        for quote in quotes:
            text = quote.xpath('.//*[@class="text"]/text()').extract_first()
            author = quote.xpath('.//*[@itemprop="author"]/text()').extract_first()
            tags = quote.xpath('.//*[@itemprop="keywords"]/@content').extract_first()

            yield{'Text':text,
                    'Author':author,
                    'Tags':tags}

        next_page_url = response.xpath('//*[@class="next"]/a/@href').extract_first()
        absolute_next_page_url = response.urljoin(next_page_url)
        yield scrapy.Request(absolute_next_page_url)

感谢所有支持！

相关问题
最新问题