Question

我正在使用下面的scrapy框架是我的spider.py代码

class Example(BaseSpider):
    name = "example"
    allowed_domains = {"http://www.example.com"}


start_urls = [
    "http://www.example.com/servlet/av/search&SiteName=page1"

def parse(self, response):
    hxs = HtmlXPathSelector(response)
    hrefs = hxs.select('//table[@class="knxa"]/tr/td/a/@href').extract()
    # href consists of all href tags and i am copying in to forwarding_hrefs by making them as a string 
    forwarding_hrefs = []
    for i in hrefs:
        forwarding_hrefs.append(i.encode('utf-8'))
    return Request('http://www.example.com/servlet/av/search&SiteName=page2',
                    meta={'forwarding_hrefs': response.meta['forwarding_hrefs']},
                   callback=self.parseJob)    


def parseJob(self, response):
    print response,">>>>>>>>>>>"

结果：

2012-07-18 17:29:15+0530 [example] DEBUG: Crawled (200) <GET http://www.example.com/servlet/av/search&SiteName=page1> (referer: None)
2012-07-18 17:29:15+0530 [MemorialReqionalHospital] ERROR: Spider error processing <GET http://www.example.com/servlet/av/search&SiteName=page2>
    Traceback (most recent call last):
      File "/usr/lib64/python2.7/site-packages/twisted/internet/base.py", line 1167, in mainLoop
        self.runUntilCurrent()
      File "/usr/lib64/python2.7/site-packages/twisted/internet/base.py", line 789, in runUntilCurrent
        call.func(*call.args, **call.kw)
      File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 361, in callback
        self._startRunCallbacks(result)
      File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 455, in _startRunCallbacks
        self._runCallbacks()
    --- <exception caught here> ---
      File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 542, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "/home/local/user/project/example/example/spiders/example_spider.py", line 36, in parse
        meta={'forwarding_hrefs': response.meta['forwarding_hrefs']},
    exceptions.KeyError: 'forwarding_hrefs'

我想要做的是从

收集所有href标签

http://www.example.com/servlet/av/search&SiteName=page1

并放入forward_hrefs并在下一个请求中调用此forward_hrefs（想要在下一个方法中使用此forward_urls列表）

http://www.example.com/servlet/av/search&SiteName=page2

我想在forward_urls中添加来自page2的href标签并在此forward_hrefs中循环并且每个href标签的yield请求，这是我的想法，但是它显示如上所述的错误，上面有什么错误代码，实际上元标记是为了复制项目。任何人都可以告诉我如何将forward_hrefs列表从parse方法复制到parseJob方法。

最后，我的目的是将forward_hrefs列表从parse方法复制到parseJob方法。

希望我解释得很抱歉，如果没有，请告诉我....

提前致谢

Answer 1

Haven没有尝试任何东西，但似乎你在这里有错误：

 return Request('http://www.example.com/servlet/av/search&SiteName=page2',
                meta={'forwarding_hrefs': response.meta['forwarding_hrefs']},
                callback=self.parseJob)

您正在传递response.meta [＆＃39; forwarding_hrefs＆＃39;]但该回复并不存在

你需要把：

 return Request('http://www.example.com/servlet/av/search&SiteName=page2',
                meta={'forwarding_hrefs': forwarding_hrefs},
                callback=self.parseJob)

因为你有forwarding_hrefs字段，这样你就可以发送它来解析meta内部的工作，然后在meta中你能够访问response.meta [＆＃39; forwarding_hrefs＆＃39;]因为它将存在于该响应对象中。

元标记在scrapy python中不起作用

1 个答案: