Question

My CrawlSpider：

class FabulousFoxSpider(CrawlSpider):
    """docstring for EventsSpider"""
    name="fabulousfox"
    allowed_domains=["fabulousfox.com"]
    start_urls=["http://www.fabulousfox.com"]
    rules = (
        Rule(SgmlLinkExtractor(
            allow=(
                '/shows_page_(single|multi).aspx\?usID=(\d)*'
                ),
            unique=True),
            'parse_fabulousfox',
            ),
        )

但是当我做scrapy crawl fabulousfox -o data.json -t json

时

我得到的输出为：

...................
......................
2014-03-01 13:11:56+0530 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
2014-03-01 13:11:56+0530 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2014-03-01 13:11:57+0530 [fabulousfox] DEBUG: Crawled (200) <GET http://www.fabulousfox.com> (referer: None)
2014-03-01 13:11:57+0530 [fabulousfox] DEBUG: Crawled (403) <GET http://www.fabulousfox.com/../shows_page_multi.aspx?usID=365> (referer: http://www.fabulousfox.com)
2014-03-01 13:11:58+0530 [fabulousfox] DEBUG: Crawled (403) <GET http://www.fabulousfox.com/../shows_page_single.aspx?usID=389> (referer: http://www.fabulousfox.com)
2014-03-01 13:11:58+0530 [fabulousfox] DEBUG: Crawled (403) <GET http://www.fabulousfox.com/../shows_page_multi.aspx?usID=388> (referer: http://www.fabulousfox.com)
2014-03-01 13:11:58+0530 [fabulousfox] DEBUG: Crawled (403) <GET http://www.fabulousfox.com/../shows_page_single.aspx?usID=394> (referer: http://www.fabulousfox.com)
2014-03-01 13:11:58+0530 [fabulousfox] DEBUG: Crawled (403) <GET http://www.fabulousfox.com/../shows_page_multi.aspx?usID=358> (referer: http://www.fabulousfox.com)
2014-03-01 13:11:58+0530 [fabulousfox] INFO: Closing spider (finished)
2014-03-01 13:11:58+0530 [fabulousfox] INFO: Dumping Scrapy stats:
    {'downloader/request_bytes': 1660,
     'downloader/request_count': 6,
     'downloader/request_method_count/GET': 6,
     'downloader/response_bytes': 12840,
     'downloader/response_count': 6,
     'downloader/response_status_count/200': 1,
     'downloader/response_status_count/403': 5,
     'finish_reason': 'finished',
     'finish_time': datetime.datetime(2014, 3, 1, 7, 41, 58, 218296),
     'log_count/DEBUG': 8,
     'log_count/INFO': 7,
     'memdebug/gc_garbage_count': 0,
     'memdebug/live_refs/FabulousFoxSpider': 1,
     'memusage/max': 33275904,
     'memusage/startup': 33275904,
     'request_depth_max': 1,
     'response_received_count': 6,
     'scheduler/dequeued': 6,
     'scheduler/dequeued/memory': 6,
     'scheduler/enqueued': 6,
     'scheduler/enqueued/memory': 6,
     'start_time': datetime.datetime(2014, 3, 1, 7, 41, 56, 360266)}
2014-03-01 13:11:58+0530 [fabulousfox] INFO: Spider closed (finished)

为什么生成的网址包含...
http://www.fabulousfox.com/../shows_page_multi.aspx?usID=365

此外，它还没有生成所有网址。这里有什么问题？

Answer 1

检查http://www.fabulousfox.com的页面HTML源代码，您会注意到这样的表行：

<tr>
    <td width="7">
        <img src="images/home_shows_frame_left.jpg" width="7" height="128" />
    </td>
    <td width="155" height="128" align="center" valign="middle">
        <a id="Box4" href="../shows_page_single.aspx?usID=394"><img id="Image4" src="../images/ShowLogos/394.jpg" alt="Rickey Smiley's" style="border-width:0px;" /></a>
    </td>
    <td width="7" align="right">
        <img src="images/home_shows_frame_right.jpg" width="7" height="128" />
    </td>
</tr>

虽然浏览器会理解这些链接并引导您http://www.fabulousfox.com/shows_page_single.aspx?usID=394，但Scrapy的SgmlLinkExtractor将在内部使用urlparse.urljoin()：

>>> import urlparse
>>> urlparse.urljoin('http://www.fabulousfox.com/', '../shows_page_single.aspx?usID=394')
'http://www.fabulousfox.com/../shows_page_single.aspx?usID=394'
>>>

您可以通过提供process_value可调用

来帮助提取链接提取器

SgmlLinkExtractor(process_value=lambda u: u.replace('../', '/'))

但在所有情况下可能都不会做你想要的事情

Answer 2

您没有正确处理相对链接。

使用urlparse.urljoin构建有效的链接。

Scrapy没有正确生成链接

2 个答案: