Question

我正在试图抓住网站ASP.NET Webform。查看源页面显示，只要页面加载，表单就会从服务器接收 VarsSessionID 。单击继续按钮后，表单会向ASMX Webserivce发送AJAX请求。 webservice将重定向返回到显示搜索结果的新URL。

我已经实现了我的scrapy蜘蛛到提交ajax post请求的程度，如图所示：

import scrapy
from scrapy.http import *
from scrapy.selector import Selector
import json
from scrapy.utils.response import open_in_browser



class TestSpider(scrapy.Spider):
    name = "test"
    allowed_domains = ['customer2.videcom.com']
    start_urls = ['http://customer2.videcom.com/med-
    view/VARS/Public/CustomerPanels/requirements.aspx?country=ng&lang=en']

def parse(self, response):
    form_data = {
        'VarsSessionID': '',
        '__VIEWSTATE': '/wEPDwULLTE4MTk4NDM5NjEPZBYCAgMPZBYCAgMPFgIeB1Zpc2libGVoZGSNuC4VK36MoPTmce49gcH1j2nxAPDYsLXii0G/syddwQ=='}
    yield FormRequest.from_response(response,
                                    formid='frmChangePage',
                                    formdata=form_data,
                                    method='POST',
                                    callback=self.after_parse,
                                    url='http://customer2.videcom.com/med-view/VARS/Public/CustomerPanels/requirements.aspx?country=ng&lang=en',
                                    )

def after_parse(self, response):
    print "====RESPONSE==="
    print response.headers
    print "=========="
    print response.request.headers
    print "=========="
    VarsSessionID = Selector(response=response).xpath("//*[@id='VarsSessionID']/@value").extract()[0]
    viewstate = Selector(response=response).xpath("//*[@id='__VIEWSTATE']/@value").extract()[0]
    print "VarsSessionID: " + VarsSessionID
    print "__VIEWSTATE: " + viewstate
    url = "http://customer2.videcom.com/med-view/VARS/Public/WebServices/AvailabilityWS.asmx/GetFlightAvailability?VarsSessionID="+VarsSessionID
    payload = {
        "FormData":
            {
                'Origin': ['LOS'],
                'VarsSessionID': VarsSessionID,
                'Destination': ['ABV'],
                'DepartureDate': ['05-May-2017'],
                'ReturnDate': '',
                'Adults': '1',
                'Children': '0',
                'SmallChildren': '0',
                "Seniors": '0',
                "Students": '0',
                "Infants": '0',
                "Youths": '0',
                "Teachers": '0',
                "SeatedInfants": '0',
                "EVoucher": '',
                "recaptcha": 'SHOW',
                "SearchUser": 'PUBLIC',
                "SearchSource": "requirements"
            }, "IsMMBChangeFlightMode": 'false'
    }
    headers = {
        'Accept': 'application/json, text/javascript, */*',
        'Accept-Encoding': 'gzip, deflate, br',
        'accept-language': 'en_US',
        'Connection': 'keep-alive',
        'content-type': 'application/json',
        'Cookie': {'VarsSessionID':''},
        'User-Agent': "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36"
    }
    yield Request(url,
                  callback=self.after_search,
                  method='POST',
                  body=json.dumps(payload),
                  headers=headers)

def after_search(self, response):
    print "========SEARCH HEADERS========"
    print response.headers
    print response.request.headers
    open_in_browser(response)

我使用Chrome开发者工具检查了标题（请求和响应），以确保Cookie和其他标题详细信息。

在运行上面的代码后，我不断获得Internal Server Error 500，如下所示：

2017-05-02 11:52:47 [scrapy.downloadermiddlewares.cookies] DEBUG: 
Sending cookies to: <POST http://customer2.videcom.com/med-view/VARS/Public/WebServices/AvailabilityWS.asmx/GetFlightAvailability?VarsSessionID=3d2048c4-2af5-4065-999f-8df6f162737b>
Cookie: ASP.NET_SessionId=v2kipt3kr2elvkat5buyajhs

2017-05-02 11:52:49 [scrapy.downloadermiddlewares.retry] DEBUG: 
Retrying <POST http://customer2.videcom.com/med-view/VARS/Public/WebServices/AvailabilityWS.asmx/GetFlightAvailability?VarsSessionID=3d2048c4-2af5-4065-999f-8df6f162737b> (failed 1 times): 500 
Internal Server Error
2017-05-02 11:52:49 [scrapy.downloadermiddlewares.cookies] DEBUG: 
Sending cookies to: <POST http://customer2.videcom.com/med-view/VARS/Public/WebServices/AvailabilityWS.asmx/GetFlightAvailability?
VarsSessionID=3d2048c4-2af5-4065-999f-8df6f162737b>
Cookie: ASP.NET_SessionId=v2kipt3kr2elvkat5buyajhs

2017-05-02 11:52:52 [scrapy.downloadermiddlewares.retry] DEBUG: 
Retrying <POST http://customer2.videcom.com/med-view/VARS/Public/WebServices/AvailabilityWS.asmx/GetFlightAvailability?
VarsSessionID=3d2048c4-2af5-4065-999f-8df6f162737b> (failed 2 times): 
500 Internal Server Error
2017-05-02 11:52:52 [scrapy.downloadermiddlewares.cookies] DEBUG: 
Sending cookies to: <POST http://customer2.videcom.com/med-view/VARS/Public/WebServices/AvailabilityWS.asmx/GetFlightAvailability?
VarsSessionID=3d2048c4-2af5-4065-999f-8df6f162737b>
Cookie: ASP.NET_SessionId=v2kipt3kr2elvkat5buyajhs

2017-05-02 11:52:54 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <POST http://customer2.videcom.com/med-view/VARS/Public/WebServices/AvailabilityWS.asmx/GetFlightAvailability?
VarsSessionID=3d2048c4-2af5-4065-999f-8df6f162737b> (failed 3 times): 
500 Internal Server Error
2017-05-02 11:52:54 [scrapy.core.engine] DEBUG: Crawled (500) <POST 
http://customer2.videcom.com/med-view/VARS/Public/WebServices/AvailabilityWS.asmx/GetFlightAvailability?
VarsSessionID=3d2048c4-2af5-4065-999f-8df6f162737b> (referer: 
http://customer2.videcom.com/med-view/VARS/Public/CustomerPanels/requirements.aspx?country=ng&lang=en)
2017-05-02 11:52:54 [scrapy.spidermiddlewares.httperror] INFO: 
Ignoring response <500 http://customer2.videcom.com/med-view/VARS/Public/WebServices/AvailabilityWS.asmx/GetFlightAvailability?
VarsSessionID=3d2048c4-2af5-4065-999f-8df6f162737b>: HTTP status code is not handled or not allowed
2017-05-02 11:52:54 [scrapy.core.engine] INFO: Closing spider 
(finished)

我需要帮助确定如何发布数据和接收搜索结果，例如我使用浏览器进行搜索时。的由于

Answer 1

将请求中的硬编码__VIEWSTATE参数替换为＆＃39; fresh＆＃39;一。

视图状态必然会在一段时间后变为无效的复杂状态。

有时在ASP网站上FormRequest.from_response没有正确捕获此参数，因此您可能需要检查response.body以了解如何提取__VIEWSTATE。

此处显示了如何执行此操作的一个很好的示例：https://blog.scrapinghub.com/2016/04/20/scrapy-tips-from-the-pros-april-2016-edition/

在Scrapy中向ASMX Webservice发送Ajax请求

1 个答案: