Question

我想从一个页面开始抓取并使用下一个url遍历到100页面，我已经在下面的代码中写了这个。我需要转到抓取中的另一个链接并提取数据并存储在项目中。我可以轻松打印要导出的所有项目数据，但无法根据需要从函数返回。

class UserLoginCrawl(CrawlSpider):
name = "mylogin"
allowed_domains = ['www.example.com']
login_page = "www.example.com/user"

start_urls = ["www.example.com/profile?page=0"]
rules = [Rule(SgmlLinkExtractor(
    allow = ('/profile\?page=\d+'),
    restrict_xpaths = ('//li[@class="pager-next"]',),canonicalize=False ),
              callback = 'parse_page',
              follow=True),]
# ulists = []

def parse_page(self, response):
    self.log ('XYZ, Started Crawling %s' %response.url)
    items = response.xpath("//div[@id='profile']/div")
    for temp in items:
        userurl = 'www.example.com'+temp.xpath("./div[@class='name']/a/@href").extract()[0]
        yield Request(url=userurl,callback=self.parse_profile_page)
    self.log ('XYZ, Finished Crawling %s' %response.url)
    # return self.ulists

def parse_profile_page(self, response):
    usritem = PostUsers()
    self.log ('XYZ, Started Crawling user Profile %s' %response.url)
    usritem["userlink"] = response.url
    usritem["fullname"] = response.xpath("//h1[@id='page-title']/text()").extract()
    relative_url = response.xpath("//div[@id='nav-content']/ul/li[2]/a/@href").extract()[0]
    usritem["postlink"] = 'www.example.com'+relative_url
    usritem["history"] = response.xpath("//div[@id='user_user_full_group_profile_main']/dl/dd[1]/text()").extract()
    # self.ulists.append(usritem)
    print usritem
    # return usritem

Answer 1

在解析方法结束时使用yield usritem。

请参阅Spider Examples

的第二个示例

如何从我的CrawlSpider返回项目？

1 个答案: