Python 3 robotparser错误

时间:2015-09-24 14:24:32

标签: python python-3.x robots.txt

我有下一个robots.txt:

User-agent: *
Disallow: /*/feed
Disallow: /*/trackback
Disallow: /category/
Disallow: /forum/
Disallow: /program/
Disallow: /wp-content/
Disallow: /trafficsystem/
Disallow: /wp-admin/
Disallow: /*?
Disallow: /*.css$
Disallow: /author/
Disallow: /*/?replytocom
Disallow: /privacy/
Disallow: /terms/
Disallow: /copyright/
Disallow: /*/users/
Disallow: /*/topic-tag/
Disallow: /quick-share/

我使用python 3.4.3和robotparser 当我将can_fetch调用到不允许的页面时,它返回True:

can_fetch("*", "http://example.com/2008/04/10/8-reasons-i-am-successful-and-you-are-not/?replytocom=154237")

为什么?

0 个答案:

没有答案