Question

我是一般的编程新手所以我希望这不是一个愚蠢的问题。我已经google'd并且花了最后4个小时试图解决这个问题，但不能真的很感激我应该尝试解决这个问题的建议/步骤。谢谢！

以下是我到目前为止蜘蛛的情况：来自scrapy.spider导入BaseSpider 来自scrapy.selector导入HtmlXPathSelector 来自tutorial.items导入TutorialItem 来自scrapy.http import FormRequest，Request

class LoginSpider(BaseSpider):
    name = 'pinterest'
    start_urls = ['https://www.pinterest.com/login/']

    def parse(self, response):
        return FormRequest.from_response(response,
                    formdata={'username_or_email': '...', 'password': '...'},
                    callback=self.after_login, dont_filter = True)

    def after_login(self, response):
        print response.url

据我所知，Scrapy自动处理cookie，因此CSRF令牌通过。我在我的设置中将COOKIES_ENABLED和COOKIES_DEBUG设置为True：

SPIDER_MIDDLEWARES = {'scrapy.contrib.downloadermiddleware.cookies.CookiesMiddleware':     
700,}
USER_AGENT = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like 
Gecko) Chrome/29.0.1547.66 Safari/537.36"
COOKIES_ENABLED = True
COOKIES_DEBUG = True

以下是调试的输出：

2013-09-27 11:11:42-0700 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2013-09-27 11:11:43-0700 [pinterest] DEBUG: Received cookies from: <200 https://
www.pinterest.com/login/>
        Set-Cookie: csrftoken=1FBJIzKqxH7XQ5tdXNtUIDHEJsL1210K; Domain=.pinteres
t.com; expires=Fri, 26-Sep-2014 18:11:46 GMT; Max-Age=31449600; Path=/
        Set-Cookie: _pinterest_sess="eJwr9UotN47SN0rUjzJ3ciwo109N8UixNPM1znK0tY8
vycxNtfUN8TXxdfEt9wsJLfdLt7VVK04tLs5MsfXMyjb0c/c0AIpX+Ia4ZfpmBeX4uqSbRFYlG0SFuFb
4ZjlWRLkHGkZWuRp6AvUBAEY1IrA="; Domain=.pinterest.com; expires=Mon, 22-Sep-2014
18:11:46 GMT; Max-Age=31103999; Path=/
2013-09-27 11:11:43-0700 [pinterest] DEBUG: Crawled (200) <GET https://www.pinte
rest.com/login/> (referer: None)
2013-09-27 11:11:43-0700 [pinterest] DEBUG: Sending cookies to: <POST https://ww
w.pinterest.com/login/>
        Cookie: csrftoken=1FBJIzKqxH7XQ5tdXNtUIDHEJsL1210K; _pinterest_sess="eJw
r9UotN47SN0rUjzJ3ciwo109N8UixNPM1znK0tY8vycxNtfUN8TXxdfEt9wsJLfdLt7VVK04tLs5MsfX
Myjb0c/c0AIpX+Ia4ZfpmBeX4uqSbRFYlG0SFuFb4ZjlWRLkHGkZWuRp6AvUBAEY1IrA="
2013-09-27 11:11:43-0700 [pinterest] DEBUG: Redirecting (302) to <GET http://www
.pinterest.com/csrf_error/> from <POST https://www.pinterest.com/login/>
2013-09-27 11:11:43-0700 [pinterest] DEBUG: Sending cookies to: <GET http://www.
pinterest.com/csrf_error/>
        Cookie: csrftoken=1FBJIzKqxH7XQ5tdXNtUIDHEJsL1210K; _pinterest_sess="eJw
r9UotN47SN0rUjzJ3ciwo109N8UixNPM1znK0tY8vycxNtfUN8TXxdfEt9wsJLfdLt7VVK04tLs5MsfX
Myjb0c/c0AIpX+Ia4ZfpmBeX4uqSbRFYlG0SFuFb4ZjlWRLkHGkZWuRp6AvUBAEY1IrA="
2013-09-27 11:11:44-0700 [pinterest] DEBUG: Crawled (200) <GET http://www.pinter
est.com/csrf_error/> (referer: https://www.pinterest.com/login/)
http://www.pinterest.com/csrf_error/

问题是在设置cookie并将其发送到登录页面后，我仍然会收到CSRF错误并被重定向。我做错了什么，我无法像浏览器那样模拟登录过程？我已经尝试将用户代理设置为iPhone并获得代码200并且没有重定向但是response.url显示“https://www.pinterest.com/login/?next=/login/”因此它仍然没有正确登录。

真的很感激我能得到的所有帮助。谢谢！

Answer 1

看起来非移动登录页面使用XHR请求来执行登录。您可以尝试深入了解XHR请求以及javascript代码，以了解您需要做什么才能在scrapy中重现请求。

但是，正如您所指出的那样，有一个移动登录页面可以通过更改用户代理来启用。

您对后一种方法的问题是电子邮件的字段是email而不是username_or_email。

这是带有微小变化的蜘蛛：

from scrapy.http import FormRequest
from scrapy.spider import BaseSpider


class LoginSpider(BaseSpider):
    name = 'pinterest'
    start_urls = ['https://www.pinterest.com/login/']
    # you can set the user agent either in the settings or the spider
    user_agent = ('Mozilla/5.0 (iPhone; CPU iPhone OS 5_0 like Mac OS X) '
                  'AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 '
                  'Mobile/9A334 Safari/7534.48.3')

    def parse(self, response):
        data = {'email': 'XXX@xxx', 'password': 'xxx'}
        # no need for dont_filter
        return FormRequest.from_response(response, formdata=data, callback=self.after_login)

    def after_login(self, response):
        print response.url

输出：

$ scrapy runspider pinterest.py
2013-09-28 19:16:58-0400 [scrapy] INFO: Scrapy 0.16.5 started (bot: scrapybot)
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMiddleware, ChunkedTransferMiddleware, DownloaderStats
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Enabled item pipelines: 
2013-09-28 19:16:58-0400 [pinterest] INFO: Spider opened
2013-09-28 19:16:58-0400 [pinterest] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2013-09-28 19:17:01-0400 [pinterest] DEBUG: Crawled (200) <GET https://www.pinterest.com/login/> (referer: None)
2013-09-28 19:17:09-0400 [pinterest] DEBUG: Redirecting (302) to <GET http://www.pinterest.com/> from <POST https://www.pinterest.com/login/?next=%2Flogin%2F>
2013-09-28 19:17:09-0400 [pinterest] DEBUG: Redirecting (302) to <GET http://www.pinterest.com/join/discover/> from <GET http://www.pinterest.com/>
2013-09-28 19:17:10-0400 [pinterest] DEBUG: Crawled (200) <GET http://www.pinterest.com/join/discover/> (referer: https://www.pinterest.com/login/)
http://www.pinterest.com/join/discover/

不接受Scrapy CSRF cookie并导致302重定向

1 个答案: