Question

我是scrapy的新手，然后我想尝试使用scrapy shell进行调试和学习，但是shell命令根本不起作用，这很奇怪。

网站似乎已成功抓取，但没有更多内容被打印出来。程序正在等待，似乎已经死了，我必须使用ctrl-c来结束它。

你可以帮忙找出错误吗？

我正在使用Anaconda + scrapy 1.0.3

$ ping 135.251.157.2

Pinging 135.251.157.2 with 32 bytes of data:
Reply from 135.251.157.2: bytes=32 time=13ms TTL=56
Reply from 135.251.157.2: bytes=32 time=14ms TTL=56
Reply from 135.251.157.2: bytes=32 time=14ms TTL=56
Reply from 135.251.157.2: bytes=32 time=14ms TTL=56

Ping statistics for 135.251.157.2:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 13ms, Maximum = 14ms, Average = 13ms


$ scrapy shell "http://135.251.157.2/"
2016-01-28 21:35:18 [scrapy] INFO: Scrapy 1.0.3 started (bot: demo)
2016-01-28 21:35:18 [scrapy] INFO: Optional features available: ssl, http11, boto
2016-01-28 21:35:18 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'demo.spiders', 'SPIDER_MODULES': ['demo.spiders'], 'LOGSTATS_INTERVAL': 0, 'BOT_NAME': 'demo'}
2016-01-28 21:35:18 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, CoreStats, SpiderState
2016-01-28 21:35:19 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, HttpProxyMiddleware, ChunkedTransferMiddleware, DownloaderStats
2016-01-28 21:35:19 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2016-01-28 21:35:19 [scrapy] INFO: Enabled item pipelines:
2016-01-28 21:35:19 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-01-28 21:35:19 [scrapy] INFO: Spider opened
2016-01-28 21:35:24 [scrapy] DEBUG: Crawled (200) <GET http://135.251.157.2/> (referer: None)
2016-01-28 21:35:24 [root] DEBUG: Using default logger
2016-01-28 21:35:24 [root] DEBUG: Using default logger
ctrl-c

$

Answer 1

我想关闭这个帖子，因为我发现根本原因与不同的终端有关。当我使用Git Bash时，它不起作用，但如果我使用Anaconda Prompt，它可以很好地工作。

[Anaconda2] D:\SVN\tools\Spider\demo>scrapy shell "http://135.251.157.2/"
2016-01-29 13:40:33 [scrapy] INFO: Scrapy 1.0.3 started (bot: demo)
2016-01-29 13:40:33 [scrapy] INFO: Optional features available: ssl, http11, boto
2016-01-29 13:40:33 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'demo.spiders', 'SPIDER_MODULES': ['demo.spiders'], 'LOGSTATS_INTERVAL': 0, 'BOT_NAME': 'demo'}
2016-01-29 13:40:33 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, CoreStats, SpiderState
2016-01-29 13:40:33 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, HttpProxyMiddleware, ChunkedTrans
ferMiddleware, DownloaderStats
2016-01-29 13:40:33 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2016-01-29 13:40:33 [scrapy] INFO: Enabled item pipelines:
2016-01-29 13:40:33 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-01-29 13:40:33 [scrapy] INFO: Spider opened
2016-01-29 13:40:41 [scrapy] DEBUG: Crawled (200) <GET http://135.251.157.2/> (referer: None)
[s] Available Scrapy objects:
[s]   crawler    <scrapy.crawler.Crawler object at 0x0136B290>
[s]   item       {}
[s]   request    <GET http://135.251.157.2/>
[s]   response   <200 http://135.251.157.2/>
[s]   settings   <scrapy.settings.Settings object at 0x034204B0>
[s]   spider     <DefaultSpider 'default' at 0x3e3c6d0>
[s] Useful shortcuts:
[s]   shelp()           Shell help (print this help)
[s]   fetch(req_or_url) Fetch request (or URL) and update local objects
[s]   view(response)    View response in a browser
2016-01-29 13:40:41 [root] DEBUG: Using default logger
2016-01-29 13:40:41 [root] DEBUG: Using default logger

In [1]:

Scrapy shell不起作用

1 个答案: