Question

尝试使用scrapy document和同事posts的代码登录CMS会员网站，但我一直在做空。我的错误消息：

2017-03-20 18:18:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://members.com/robots.txt> (referer: None)
2017-03-20 18:18:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://members.com/wp-login.php> (referer: None)
2017-03-20 18:18:07 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <POST http://members.com/login.php> from <POST http://members.com/login.ph
p?wpe-login=membersipa>

我尝试将用户代理更改为：

USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; rv:32.0) Gecko/20100101 Firefox/32.0'

但我的错误是：

2017-03-20 17:47:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET     http://members.com/robots.txt> (referer: None)
2017-03-20 17:47:22 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://members.com/wp-login.php> (referer: None)
2017-03-20 17:47:23 [scrapy.core.engine] DEBUG: Crawled (403) <POST http://members.com/wp-login.php?wpe-login=membersipa> (referer: http://members.com/wp-login.php)
2017-03-20 17:47:23 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 http://members.com/wp-login.php?wpe-login=membersipa>: HTTP status code is not handled or not
allowed

这是产生错误的代码：

import scrapy

class LoginSpider(scrapy.Spider):
    name = 'freddy'
    start_urls = ['http://members.com/wlogin.php']

    def parse(self, response):
        return scrapy.FormRequest.from_response(
            response,
            formdata={'log': 'name', 'pwd': 'password'},
            callback=self.after_login
        )

    def after_login(self, response):
        # check login succeed before going on
        if "authentication failed" in response.body:
            self.logger.error("Login failed")
            return
    else:
            return Request(url="http://members.com",
               callback=self.parse_ipro)

    def parse_ipro(self, response):
        title = response.xpath('/html/body/div[2]/div/div[1]/div/div/div[2]/div/div/main/article/header/h1').extract_first()
        with open(filename, 'wb') as f:
            f.write(response.body)
        self.log('Saved file %s' % filename)

最终，我想使用scrapy shell来测试选择器并尝试使用scrapy

尝试用scrapy shell但也敲了屁股：

import scrapy

class LoginSpider(scrapy.Spider):
    name = 'freddy'
    start_urls = ['http://members.com/wlogin.php']

    def parse(self, response):
        return scrapy.FormRequest.from_response(
            response,
            formdata={'log': 'name', 'pwd': 'password'},
            callback=self.after_login
        )

    def after_login(self, response):
        # check login succeed before going on
        if "authentication failed" in response.body:
            self.logger.error("Login failed")
            return

在shell中测试了这个：

response.xpath('//title/text()').extract_first()

但收到301和302重定向

添加后：

def parse(self, response):
    return scrapy.FormRequest.from_response(
        response,
        headers={'Content-Type': 'text/html; charset=UTF-8', 'User-Agent':
                 'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0'},
        formdata={'log': 'Name', 'pwd': 'Password', },
        callback=self.after_login
    )

邮件更改为：

2017-03-22 03:46:07 [scrapy.core.engine] INFO: Spider opened
2017-03-22 03:46:07 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-03-22 03:46:07 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 
2017-03-22 03:46:08 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://members.com/login.php> (referer: None)
2017-03-22 03:46:08 [scrapy.core.scraper] ERROR: Spider error processing <GET http://members.com/login.php> (referer: None)
Traceback (most recent call last):

感谢帮助

Answer 1

您很可能遗漏了FormRequest中的某些标题。

在浏览器工具中打开网络标签，找到您要查找的请求，并查看“请求标头”部分（请参阅相关问题Can scrapy be used to scrape dynamic content from websites that are using AJAX?）。有些标题不是必需的，有些标题已包含在FormRequest中，但有些标题不是这样，您需要复制它们。

通常需要复制Content-Type标头。

headers = {
    'Content-Type': 'json/...',
}
req = FormRequest(url, formdata=form, headers=headers)

在FormRequest中使用Scrapy Shell

1 个答案: