如何从收到的响应中抓取HTML代码？

Question

我正在尝试抓取和飞溅网站。 我想从图像中显示的响应中抓取特定的html代码。 这是带有标题的响应：

这是响应（我要抓取的html）：

response.body返回页面的源代码，而没有我上面提到的响应中需要的html代码。

import scrapy
from scrapy_splash import SplashRequest
from bs4 import BeautifulSoup

class NetherSplashSpider(scrapy.Spider):
    name = 'nether_splash'
    download_delay = 10

    custom_settings = {
        'SPLASH_URL': 'http://localhost:8050',
        'DOWNLOADER_MIDDLEWARES': {
            'scrapy_splash.SplashCookiesMiddleware': 723,
            'scrapy_splash.SplashMiddleware': 725,
            'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
        },
        'SPIDER_MIDDLEWARES': {
            'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
        },
        'DUPEFILTER_CLASS': 'scrapy_splash.SplashAwareDupeFilter',
    }

    def start_requests(self):
        yield SplashRequest(
            url='https://www.gaslicht.com/stroom-vergelijken?partial=true&aanbieders=eneco&skip=0&take=10&_=1559207102962',
            callback=self.parse,
        )


    def parse(self, response):



        filename = 'splash.html'
        with open(filename, 'wb') as f:
            f.write(response.body)

Answer 1

为了加载整个页面，您将需要添加“ wait”参数。 尝试在您的SplashRequest中添加“ args = {'wait'：1.0}”。

yield SplashRequest(
            url='https://www.gaslicht.com/stroom-vergelijken?partial=true&aanbieders=eneco&skip=0&take=10&_=1559207102962',
            callback=self.parse, args={'wait': 1.0}
        )