我正试图从亚马逊网站上搜集一些视频信息：
https://www.amazon.com/s/ref=nb_sb_noss_2?url=search-alias%3Dinstant-video&field-keywords=video
我开始了一个项目然后生成蜘蛛 scrapy startproject amazon_books_crawler
scrapy genspider amazon_books_crawler 'https://www.amazon.com/s/ref=nb_sb_noss_2?url=search-alias%3Dinstant-video&field-keywords=video'
以下是我amazon_books_crawler.py
# -*- coding: utf-8 -*-
import scrapy
from scrapy import Request
class AmazonBooksCrawlerSpider(scrapy.Spider):
name = "amazon_books_crawler"
allowed_domains = ["https://www.amazon.com"]
start_urls = ['https://www.amazon.com/s/ref=nb_sb_noss_2?url=search-alias%3Dinstant-video&field-keywords=video']
custom_settings = {
'USER_AGENT': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36',
# 'FEED_EXPORT_FIELDS': ['title', 'url', 'address', 'compensation', 'employment', 'description'],
'DOWNLOAD_DELAY': 1,
'ROBOTSTXT_OBEY': False
}
def parse(self, response):
items = response.xpath("//ul[contains(@id, 's-results-list-atf')]/li")
for item in items:
title = item.xpath(" .//a[@class='a-link-normal s-access-detail-page s-color-twister-title-link a-text-normal']/@title").extract_first()
video_url = item.xpath(".//a[@class='a-link-normal s-access-detail-page s-color-twister-title-link a-text-normal']/@href").extract_first()
asin = item.xpath("./@data-asin").extract_first()
yield ({'title': title,
'asin': asin,
'url': video_url})
然而，当我开始爬行时，似乎Scrapy只从网站上抓下4件物品并停止了 我不知道它为什么在它只获得4件后停止了。谁能帮忙看看我在编码时做错了什么？我花了几个小时但却想不出来......
下面是我得到的输出：
2018-01-29 08:49:47 [scrapy.utils.log] INFO: Scrapy 1.3.0 started (bot: amazon_books)
2018-01-29 08:49:47 [scrapy.utils.log] INFO: Overridden settings: {'FEED_URI': 'temp.csv', 'NEWSPIDER_MODULE': 'amazon_books.spiders', 'BOT_NAME': 'amazon_books', 'FEED_FORMAT': 'csv', 'SPIDER_MODULES': ['amazon_books.spiders']}
2018-01-29 08:49:47 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.feedexport.FeedExporter',
'scrapy.extensions.logstats.LogStats']
2018-01-29 08:49:47 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-01-29 08:49:47 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-01-29 08:49:47 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2018-01-29 08:49:47 [scrapy.core.engine] INFO: Spider opened
2018-01-29 08:49:47 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-01-29 08:49:47 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-01-29 08:49:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.com/s/ref=nb_sb_noss_2?url=search-alias%3Dinstant-video&field-keywords=video> (referer: None)
2018-01-29 08:49:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.com/s/ref=nb_sb_noss_2?url=search-alias%3Dinstant-video&field-keywords=video>
{'title': 'Arrival', 'asin': 'B01M2C4NP8', 'url': 'https://www.amazon.com/Arrival-Amy-Adams/dp/B01M2C4NP8/ref=sr_1_1/135-2369526-7990427?s=instant-video&ie=UTF8&qid=1517233787&sr=1-1&keywords=video'}
2018-01-29 08:49:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.com/s/ref=nb_sb_noss_2?url=search-alias%3Dinstant-video&field-keywords=video>
{'title': 'Absentia Season 1', 'asin': 'B078KNYFR4', 'url': 'https://www.amazon.com/Season-1-Offical-Trailer/dp/B0789DVY2J/ref=sr_1_2/135-2369526-7990427?s=instant-video&ie=UTF8&qid=1517233787&sr=1-2&keywords=video'}
2018-01-29 08:49:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.com/s/ref=nb_sb_noss_2?url=search-alias%3Dinstant-video&field-keywords=video>
{'title': 'Dandelion Dead', 'asin': 'B01N0XXQP5', 'url': 'https://www.amazon.com/Part-1/dp/B01MT68VT6/ref=sr_1_3/135-2369526-7990427?s=instant-video&ie=UTF8&qid=1517233787&sr=1-3&keywords=video'}
2018-01-29 08:49:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.com/s/ref=nb_sb_noss_2?url=search-alias%3Dinstant-video&field-keywords=video>
{'title': 'Dunkirk: The Real Story', 'asin': 'B074MGB6M3', 'url': 'https://www.amazon.com/Dunkirk-Real-Story-Bruce-Vigar/dp/B074MGB6M3/ref=sr_1_4/135-2369526-7990427?s=instant-video&ie=UTF8&qid=1517233787&sr=1-4&keywords=video'}
2018-01-29 08:49:48 [scrapy.core.engine] INFO: Closing spider (finished)
2018-01-29 08:49:48 [scrapy.extensions.feedexport] INFO: Stored csv feed (4 items) in: temp.csv
2018-01-29 08:49:48 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 351,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 107102,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2018, 1, 29, 13, 49, 48, 563901),
'item_scraped_count': 4,
'log_count/DEBUG': 6,
'log_count/INFO': 8,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2018, 1, 29, 13, 49, 47, 378678)}
2018-01-29 08:49:48 [scrapy.core.engine] INFO: Spider closed (finished)
答案 0 :(得分：1)
您的准则是正确的，您被列入黑名单。 你应该使用这种技术来防止这种情况，比如