Scrapy抓取工具无法抓取或无法写入csv文件？

时间：2017-01-09 22:57:43

标签： python python-2.7 web-scraping scrapy web-crawler

项目：https://github.com/smalik/resume_scraper

我使用以下命令运行此bot：

sudo scrapy crawl indeed_resume -o items.csv

问题是它没有正确抓取（0页）或由于某种原因无法写入csv文件。错误可能是由于模块折旧造成的吗？这是运行时的终端输出：

/home/shap/Desktop/resume_crawlers/resume_scraper-master/resume_data/spiders/resume_indeed.py:10: ScrapyDeprecationWarning: Module `scrapy.contrib.spiders` is deprecated, use `scrapy.spiders` instead
from scrapy.contrib.spiders import CrawlSpider, Rule
/home/shap/Desktop/resume_crawlers/resume_scraper-master/resume_data/spiders/resume_indeed.py:11: ScrapyDeprecationWarning: Module `scrapy.contrib.linkextractors` is deprecated, use `scrapy.linkextractors` instead
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
/home/shap/Desktop/resume_crawlers/resume_scraper-master/resume_data/spiders/resume_indeed.py:11: ScrapyDeprecationWarning: Module `scrapy.contrib.linkextractors.sgml` is deprecated, use `scrapy.linkextractors.sgml` instead
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
/home/shap/Desktop/resume_crawlers/resume_scraper-master/resume_data/spiders/resume_indeed.py:33: ScrapyDeprecationWarning: SgmlLinkExtractor is deprecated and will be removed in future releases. Please use scrapy.linkextractors.LinkExtractor
rules = (Rule (SgmlLinkExtractor(restrict_xpaths = ('//a[@class="instl confirm-nav next"]')), callback = "parse_item", follow = True),)
2017-01-09 14:39:22 [scrapy.utils.log] INFO: Scrapy 1.3.0 started (bot: resume_data)
2017-01-09 14:39:22 [scrapy.utils.log] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'resume_data.spiders', 'FEED_URI': 'items.csv', 'DEPTH_LIMIT': 50, 'SPIDER_MODULES': ['resume_data.spiders'], 'BOT_NAME': 'resume_data', 'AJAXCRAWL_ENABLED': True, 'COOKIES_ENABLED': False, 'FEED_FORMAT': 'csv', 'MEMUSAGE_REPORT': True, 'DOWNLOAD_DELAY': 2}
2017-01-09 14:39:22 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.feedexport.FeedExporter',
'scrapy.extensions.logstats.LogStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.corestats.CoreStats']
2017-01-09 14:39:22 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-01-09 14:39:22 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-01-09 14:39:22 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-01-09 14:39:22 [scrapy.core.engine] INFO: Spider opened
2017-01-09 14:39:22 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-01-09 14:39:22 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-01-09 14:39:22 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (307) to <GET https://www.indeed.com/resumes/mechanical-engineer> from <GET http://www.indeed.com/resumes/mechanical-engineer>
2017-01-09 14:39:24 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (307) to <GET https://www.indeed.com/resumes/mechanical-engineering> from <GET http://www.indeed.com/resumes/mechanical-engineering>
2017-01-09 14:39:26 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (307) to <GET https://www.indeed.com/resumes/piping-engineer> from <GET http://www.indeed.com/resumes/piping-engineer>
2017-01-09 14:39:28 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (307) to <GET https://www.indeed.com/resumes/java> from <GET http://www.indeed.com/resumes/java>
2017-01-09 14:39:30 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (307) to <GET https://www.indeed.com/resumes/SAS> from <GET http://www.indeed.com/resumes/SAS>
2017-01-09 14:39:32 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (307) to <GET https://www.indeed.com/resumes/javascript> from <GET http://www.indeed.com/resumes/javascript>
2017-01-09 14:39:35 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (307) to <GET https://www.indeed.com/resumes/marketing-analyst> from <GET http://www.indeed.com/resumes/marketing-analyst>
2017-01-09 14:39:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.indeed.com/resumes/mechanical-engineering> (referer: None)
2017-01-09 14:39:43 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.indeed.com/resumes/piping-engineer> (referer: None)
2017-01-09 14:39:45 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.indeed.com/resumes/java> (referer: None)
2017-01-09 14:39:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.indeed.com/resumes/SAS> (referer: None)
2017-01-09 14:39:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.indeed.com/resumes/javascript> (referer: None)
2017-01-09 14:39:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.indeed.com/resumes/marketing-analyst> (referer: None)
2017-01-09 14:40:00 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.indeed.com/resumes/mechanical-engineer> (referer: None)
2017-01-09 14:40:00 [scrapy.core.engine] INFO: Closing spider (finished)
2017-01-09 14:40:00 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 3260,
'downloader/request_count': 14,
'downloader/request_method_count/GET': 14,
'downloader/response_bytes': 127215,
'downloader/response_count': 14,
'downloader/response_status_count/200': 7,
'downloader/response_status_count/307': 7,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2017, 1, 9, 22, 40, 0, 662954),
'log_count/DEBUG': 15,
'log_count/INFO': 7,
'response_received_count': 7,
'scheduler/dequeued': 14,
'scheduler/dequeued/memory': 14,
'scheduler/enqueued': 14,
'scheduler/enqueued/memory': 14,
'start_time': datetime.datetime(2017, 1, 9, 22, 39, 22, 288258)}
2017-01-09 14:40:00 [scrapy.core.engine] INFO: Spider closed (finished)

1 个答案:

答案 0 :(得分：0)

蜘蛛似乎不会抓取除start_urls中定义的网址以外的任何内容。

您的代码中的问题是，Rules和Linkextractors没有提取任何内容 测试linkextractors的一个好方法是使用scrapy shell：

$ scrapy shell "https://www.indeed.com/resumes/javascript"
from scrapy.linkextractors import LinkExtractor
le = LinkExtractor(restrict_xpaths='//a[contains(@class,"app_link")]')
le.extract_links(response)
#< returns 50 scrapy.link.Link objects here>

上面的示例包含用于捕获工作机会网址的正确xpath。您可以使用此方法修复蜘蛛。

相关问题
最新问题