Question

我打开scrapy shell如下

scrapy shell "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/"

这给了我：

[s] Available Scrapy objects:
[s]   hxs        <HtmlXPathSelector xpath=None data=u'<html><head><meta http-equiv="Content-Ty'>
[s]   item       {}
[s]   request    <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
[s]   response   <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
[s]   settings   <CrawlerSettings module=None>
[s]   spider     <BaseSpider 'default' at 0x9e1d3ec>
[s] Useful shortcuts:
[s]   shelp()           Shell help (print this help)
[s]   fetch(req_or_url) Fetch request (or URL) and update local objects
[s]   view(response)    View response in a browser


In [1]: hxs.select('//title')
Out[1]: [<HtmlXPathSelector xpath='//title' data=u'<title>Open Directory - Computers: Progr'>]

回复的标题符合预期：

In [1]: hxs.select('//title')
Out[1]: [<HtmlXPathSelector xpath='//title' data=u'<title>Open Directory - Computers: Progr'>]

现在我通过一个简单的提取来跟进：

In [2]: fetch("http://www.google.com")

shell输出表明对象已更新：

In [2]: fetch("http://www.google.com")
2013-10-18 23:10:09+0530 [default] DEBUG: Redirecting (302) to <GET http://www.google.co.in/?gws_rd=cr&ei=eHJhUo2sOobSrQeM5ICAAg> from <GET http://www.google.com>
2013-10-18 23:10:09+0530 [default] DEBUG: Crawled (200) <GET http://www.google.co.in/?gws_rd=cr&ei=eHJhUo2sOobSrQeM5ICAAg> (referer: None)
[s] Available Scrapy objects:
[s]   hxs        <HtmlXPathSelector xpath=None data=u'<html itemscope="" itemtype="http://sche'>
[s]   item       {}
[s]   request    <GET http://www.google.com>
[s]   response   <200 http://www.google.co.in/?gws_rd=cr&ei=eHJhUo2sOobSrQeM5ICAAg>
[s]   settings   <CrawlerSettings module=None>
[s]   spider     <BaseSpider 'default' at 0x9e1d3ec>
[s] Useful shortcuts:
[s]   shelp()           Shell help (print this help)
[s]   fetch(req_or_url) Fetch request (or URL) and update local objects
[s]   view(response)    View response in a browser

然而，我发现他们没有。 view（响应）显示了dmoz页面

提取标题给出了同样的旧标题：

In [3]: hxs.select('//title')
Out[3]: [<HtmlXPathSelector xpath='//title' data=u'<title>Open Directory - Computers: Progr'>]

我在这里缺少什么？

谢谢！

Answer 1

即使它对我有用，也许它与这个错误相关：https://github.com/scrapy/scrapy/issues/396

您可以尝试运行最新的开发版本吗？

更新：问题来自ipython 0.10，更新到最新版本，它应该按预期工作。它也固定在scrapy 0.19+（最新开发）。所以你可以升级ipython或scrapy。

scrapy shell中的fetch不会更新对象。我在这里缺少什么？

1 个答案: