Question

我试图从亚马逊中搜索一个类别，但我在Scrapy中获得的链接与浏览器中的链接不同。现在我正在尝试按照下一页跟踪并在Scrapy中打印（将response.body打印到txt文件中）我看到了这些链接：

<span class="pagnMore">...</span>
<span class="pagnLink"><a href="/s?ie=UTF8&page=4&rh=n%3A2619533011%2Ck%3Apet%20supplies%2Cp_72%3A2661618011%2Cp_n_date_first_available_absolute%3A2661609011" >4</a></span>
<span class="pagnCur">5</span>
<span class="pagnLink"><a href="/s?ie=UTF8&page=6&rh=n%3A2619533011%2Ck%3Apet%20supplies%2Cp_72%3A2661618011%2Cp_n_date_first_available_absolute%3A2661609011" >6</a></span>
<span class="pagnMore">...</span>
<span class="pagnDisabled">20</span>
<span class="pagnRA"> <a title="Next Page"
                   id="pagnNextLink"
                   class="pagnNext"
                   href="/s?ie=UTF8&page=6&rh=n%3A2619533011%2Ck%3Apet%20supplies%2Cp_72%3A2661618011%2Cp_n_date_first_available_absolute%3A2661609011">
<span id="pagnNextString">Next Page</span>

我想关注pagnNextString链接，但我的蜘蛛甚至没有开始抓取：

Rule(SgmlLinkExtractor(allow=("n\%3A2619533011\%", ),restrict_xpaths=('//*[@id="pagnNextLink"]',)) , callback="parse_items", follow= True),

如果我摆脱了规则或做某事。就像'^http.*'它正在工作但它遵循一切。我在这里做错了什么？

Answer 1

请尝试仅检查page参数：

Rule(SgmlLinkExtractor(allow=r"page=\d+"), callback="parse_items", follow= True),

Scrapy LinkExtractor - RegEx遵循哪个？

1 个答案: