Question

我正在尝试从以下网站中摘录：https://www.mrlodge.de/wohnungen/ 蜘蛛可以正常工作，但不会对我的有效载荷进行分页。我只会回到同一页面。我尝试使用json库配置有效负载，但是有效负载本身不在json中。请帮忙。

import scrapy

class MrlodgeSpiderSpider(scrapy.Spider):
    name = 'mrlodge_spider'

    def start_requests(self):
        for pageNumber in range(1,10):
            s = """mrl_ft%5Bfd%5D%5Bdate_from%5D=&mrl_ft%5Bfd%5D%5Brent_from%5D=1000&mr\
                l_ft%5Bfd%5D%5Brent_to%5D=8500&mrl_ft%5Bfd%5D%5Bpersons%5D=1&mrl_ft%5Bfd\
                %5D%5Bkids%5D=0&mrl_ft%5Bfd%5D%5Brooms_from%5D=1&mrl_ft%5Bfd%5D%5Brooms_to\
                %5D=9&mrl_ft%5Bfd%5D%5Barea_from%5D=20&mrl_ft%5Bfd%5D%5Barea_to%5D=480&\
                mrl_ft%5Bfd%5D%5Bsterm%5D=&mrl_ft%5Bfd%5D%5Bradius%5D=50&mrl_ft%5Bfd\
                %5D%5Bmvv%5D=&mrl_ft%5Bfd%5D%5Bobjecttype_cb%5D%5B%5D=w&mrl_ft%5B\
                fd%5D%5Bobjecttype_cb%5D%5B%5D=h&mrl_ft%5Bpage%5D={}""".format(pageNumber)
            payload = '{'+s+'}'
            print(payload)
            yield scrapy.Request(url='https://www.mrlodge.de/wohnungen/', method='POST',
                                body=payload, headers={'content-type': 'application/json'})


    def parse(self, response):
        for apartment in response.xpath("//div[@class='mrl-ft-results mrlobject-list']/div"):
            yield {
                'info': apartment.xpath(".//div[@class='obj-smallinfo']/text()").get()
                 }

Answer 1

我看到标题错误且不完整。我看到标题如下：

Host: www.mrlodge.de
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:67.0) Gecko/20100101 Firefox/67.0
Accept: application/json, text/javascript, */*; q=0.01
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://www.mrlodge.de/wohnungen/
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
X-Requested-With: XMLHttpRequest
Content-Length: 468
Connection: keep-alive
Cookie: fe_typo_user=cbf861eb412b7182ccf72aa5ca87c932; uac=true
Pragma: no-cache
Cache-Control: no-cache

Content-Type也不同。

Answer 2

以下方法对我有用。

Dim rng1 As Range
Dim rng2 As Range
Dim myavg As Range

Set rng1 = Range("B5")

Do While rng1.Value <> ""
    Set rng2 = Cells(Rows.count, rng1.Column).End(xlUp) ' using @Chronocidal's suggestion above
    Set myavg = rng2.Offset(1, 0)

    myavg.Formula = "=Average(" & rng1.Address & ":" & rng2.Address & ")"

    Set rng1 = rng1.Offset(0, 1)
Loop

当然，您必须动态设置数据中的某些值（例如mrl_ft [page]）才能浏览所有页面。

Scrapy：分页与发布请求不起作用

2 个答案: