Question

我正在尝试构建只用Scrapy破坏动态验证码的蜘蛛，但是我做到了，但是当我打破验证码时并不总是正确的，所以我必须使其重试多次（最大10）真正进入多个帐户的“登录”页面（顺便说一句，我正在使用PIL和pytesseract处理验证码）。我已经读过How do I set up Scrapy to deal with a captcha，这对我有很大帮助，但会给我这个错误：

TypeError: process_response() got multiple values for argument 'request'

这是我的蜘蛛代码：

#self.crawler.settings['MY_ACCOUNTS']= i have defined my list of accounts with "keys"

def start_requests(self):
    requests = []

    for i, account in enumerate(self.crawler.settings['MY_ACCOUNTS']):
        request = scrapy.Request(same_url, callback = self.parse, meta={'key':account['key']} ,dont_filter=True)
        request.meta['cookiejar'] = i
        requests.append(request)
        yield request

#I HAVE TO USE MULTIPLE REQUEST JUST TO GET THE CORRECT SESSION FOR THE COOKIES AND GET THE IMAGE, IF NOT THE IMAGE ALWAYS CHANGE
def parse(self, response):
    key= response.meta.get('key')
    sessionid = response.headers.getlist('Set-Cookie')[0].decode().split(';')[0]
    cookien = sessionid.split('=')[0].strip()
    sessionid = sessionid.split('=')[1].strip()
    yield scrapy.Request(same_url, meta = {'cookieN': cookien, 'cookieV' : sessionid, 'key':key,'cookiejar': response.meta['cookiejar']},cookies = {cookien :sessionid}, callback = self.parse_with_cookies, dont_filter=True)

#HERE I'M TRYING TO BREAK THE CAPTCHA FILLING A FORM
def parse_with_cookies(self, response):
    #break captcha (get text)

    data = {'key':key,
    'catcha':text
    }#fill with the data

    yield FormRequest.from_response(response=response,formname="mainForm", formdata=data, callback = self.parse_each_key,meta = {'solve_captcha': True,'CookieName':cookieN,'CookieValue':cookieV , 'key':key,'cookiejar': response.meta['cookiejar'],'catpcha_retries':0},cookies={cookieN:cookieV},dont_filter = True)

#the status code is always 200 even if the captcha is not broken so i search for words if it's incorrect
def parse_each_key(self, response):
    if ("incorrect" not in response.text) and ("error" not in response.text):
         #extract what i want
    else:
         self.logger.error("Login failed")

----- MIDDLEWARE ---

#i haven't change to much from the link, i know that i have to fill the functions find_catpcha and solve_captcha but it doesn't even get there

class CaptchaMiddleware(object):
max_retries = 5
def process_response(request, response, spider):
    if not request.meta.get('solve_captcha', False):
        return response  # only solve requests that are marked with meta key
    catpcha = find_catpcha(response)
    solved = solve_captcha(captcha)
    if solved:
        response.meta['catpcha'] = captcha
        response.meta['solved_catpcha'] = solved
        return response
    else:
        # retry page for new captcha
        # prevent endless loop
        if request.meta.get('catpcha_retries', 0) == 5:
            logging.warning('max retries for captcha reached for {}'.format(request.url))
            raise IgnoreRequest 
        request.meta['dont_filter'] = True
        request.meta['captcha_retries'] = request.meta.get('captcha_retries', 0) + 1
        return request

我不知道这是因为我同时发送多个请求还是代码中有某些内容。

谢谢

Answer 1

您在中间件中self的定义中缺少process_response()参数。这是造成异常的原因。

如果将process_response(request, response, spider)替换为process_response(self, request, response, spider)，则应停止获取该异常。

如何在SCRAPY中处理对MIDDLEWARE的多次请求（验证码多次重试）

1 个答案: