scrapy 301重定向亚马逊

时间:2016-08-01 07:05:39

标签: python scrapy scrapy-shell

我试图跟随scrapy教程但是以亚马逊为例,而不是在他们的在线教程中使用dmoz。

当我运行以下命令时,我注意到我得到301重定向。

scrapy shell "https://www.amazon.com/gp/product/B00KLTPUU0"

2016-07-31 23:36:13 [scrapy] INFO: Scrapy 1.1.1 started (bot: scrapybot)
2016-07-31 23:36:13 [scrapy] INFO: Overridden settings: {'LOGSTATS_INTERVAL': 0, 'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter'}
2016-07-31 23:36:13 [scrapy] INFO: Enabled extensions:['scrapy.extensions.telnet.TelnetConsole','scrapy.extensions.corestats.CoreStats']
2016-07-31 23:36:13 [scrapy] INFO: Enabled downloader middlewares:['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware','scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware','scrapy.downloadermiddlewares.useragent.UserAgentMiddleware','scrapy.downloadermiddlewares.retry.RetryMiddleware','scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware','scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware','scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware','scrapy.downloadermiddlewares.redirect.RedirectMiddleware','scrapy.downloadermiddlewares.cookies.CookiesMiddleware','scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware','scrapy.downloadermiddlewares.stats.DownloaderStats']
2016-07-31 23:36:13 [scrapy] INFO: Enabled spider middlewares:['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware','scrapy.spidermiddlewares.offsite.OffsiteMiddleware','scrapy.spidermiddlewares.referer.RefererMiddleware','scrapy.spidermiddlewares.urllength.UrlLengthMiddleware','scrapy.spidermiddlewares.depth.DepthMiddleware']
2016-07-31 23:36:13 [scrapy] INFO: Enabled item pipelines:[]
2016-07-31 23:36:13 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-07-31 23:36:13 [scrapy] INFO: Spider opened
2016-07-31 23:36:17 [scrapy] DEBUG: Crawled (301) <GET https://www.amazon.com/gp/product/B00KLTPUU0> (referer: None)
2016-07-31 23:36:17 [root] DEBUG: Using default logger
2016-07-31 23:36:17 [root] DEBUG: Using default logger
[s] Available Scrapy objects:
[s]   crawler    <scrapy.crawler.Crawler object at 0x7fb02b791f50>
[s]   item       {}
[s]   request    <GET https://www.amazon.com/gp/product/B00KLTPUU0>
[s]   response   <301 https://www.amazon.com/gp/product/B00KLTPUU0>
[s]   settings   <scrapy.settings.Settings object at 0x7fb02b7919d0>
[s]   spider     <DefaultSpider 'default' at 0x7fb02b117510>
[s] Useful shortcuts:
[s]   shelp()           Shell help (print this help)
[s]   fetch(req_or_url) Fetch request (or URL) and update local objects
[s]   view(response)    View response in a browser

任何人都可以解释为什么我使用scrapy获得这个亚马逊页面的301重定向?

0 个答案:

没有答案