Question

网址我试图抓取：https://www.uvic.ca/BAN1P/bwckschd.p_disp_dyn_sched

共有3页，第一页用于选择字词，第二页用于选择主题，以及包含实际课程信息的页面。

我遇到的问题是，一旦 subject（）调用 courses（），就会回复写入该文件的response.body中的html主题页面的html而不是课程页面。如何告诉我发送正确的表单数据以便收到正确的答复？

# term():
#   Selects the school term to use. Clicks submit

def term(self, response):
    return scrapy.FormRequest.from_response(
    response,
    formxpath="/html/body/div[3]/form",
    formdata={
        "p_term" : "201705" },
    clickdata = { "type": "submit" },
    callback=self.subject
    )

# subject():
#   Selects the subject to query. Clicks submit

def subject(self, response):
    return scrapy.FormRequest.from_response(
    response,
    formxpath="/html/body/div[3]/form",
    formdata={
        "sel_subj" : "ART" },
    clickdata = { "type": "submit" },
    callback=self.courses
    )

# courses():
#   Currently just saves all the html on the page.

def courses(self, response):
    page = response.url.split("/")[-1]
    filename = 'uvic-%s.html' % page
    with open(filename, 'wb') as f:
        f.write(response.body)
    self.log('Saved file %s' % filename)

调试输出

2017-04-02 01:15:28 [scrapy.utils.log] INFO: Scrapy 1.3.3 started (bot: scrapy4uvic)
2017-04-02 01:15:28 [scrapy.utils.log] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'scrapy4uvic.spiders', 'SPIDER_MODULES': ['scrapy4uvic.spiders'], 'ROBOTSTXT_OBEY': True, 'BOT_NAME': 'scrapy4uvic'}
2017-04-02 01:15:28 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats']
2017-04-02 01:15:28 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-04-02 01:15:28 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-04-02 01:15:28 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-04-02 01:15:28 [scrapy.core.engine] INFO: Spider opened
2017-04-02 01:15:28 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-04-02 01:15:28 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-04-02 01:15:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.uvic.ca/robots.txt> (referer: None)
2017-04-02 01:15:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.uvic.ca/BAN1P/bwckschd.p_disp_dyn_sched> (referer: None)
2017-04-02 01:15:29 [scrapy.core.engine] DEBUG: Crawled (200) <POST https://www.uvic.ca/BAN1P/bwckgens.p_proc_term_date> (referer: https://www.uvic.ca/BAN1P/bwckschd.p_disp_dyn_sched)
2017-04-02 01:15:29 [scrapy.core.engine] DEBUG: Crawled (200) <POST https://www.uvic.ca/BAN1P/bwckschd.p_get_crse_unsec> (referer: https://www.uvic.ca/BAN1P/bwckgens.p_proc_term_date)
2017-04-02 01:15:30 [uvic] DEBUG: Saved file uvic-bwckschd.p_get_crse_unsec.html
2017-04-02 01:15:30 [scrapy.core.engine] INFO: Closing spider (finished)
2017-04-02 01:15:30 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 2335,
 'downloader/request_count': 4,
 'downloader/request_method_count/GET': 2,
 'downloader/request_method_count/POST': 2,
 'downloader/response_bytes': 105499,
 'downloader/response_count': 4,
 'downloader/response_status_count/200': 4,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2017, 4, 2, 8, 15, 30, 103536),
 'log_count/DEBUG': 6,
 'log_count/INFO': 7,
 'request_depth_max': 2,
 'response_received_count': 4,
 'scheduler/dequeued': 3,
 'scheduler/dequeued/memory': 3,
 'scheduler/enqueued': 3,
 'scheduler/enqueued/memory': 3,
 'start_time': datetime.datetime(2017, 4, 2, 8, 15, 28, 987034)}
2017-04-02 01:15:30 [scrapy.core.engine] INFO: Spider closed (finished)

Answer 1

好像你遗漏了subject()中的一些表单数据。

我设法让它与之合作：

formdata={
    "sel_subj": ["dummy", "ART"],
}

我是如何进行调试的首先，您不必保存到文件，您可以在抓取期间inspect_response：

def courses(self, response):
    from scrapy.shell import inspect_response
    inspect_response(response, self)

这会打开一个包含response和request对象的shell，您甚至可以调用view(response)在浏览器中打开html。如果可用，它还会使用ipython或bpython个shell，在下面的示例中，我使用ipython进行方便的格式化。

其次我检查了我的浏览器（firefox）当我点击按钮并将其复制到变量bar下的shell并将其与scrapy发送的请求主体进行比较时，它发送的是什么形式：

bar = '''term_in=201705&sel_subj=dummy&sel_day=dummy&sel_schd=dummy&sel_insm=dummy&
      sel_camp=dummy&sel_levl=dummy                                                      
      &sel_sess=dummy&sel_instr=dummy&sel_ptrm=dummy&sel_attr=dummy&sel_subj=ART&sel_crse
      =&sel_title=&sel_schd                                                              
      =%25&sel_insm=%25&sel_from_cred=&sel_to_cred=&sel_camp=%25&sel_levl=%25&sel_ptrm=%2
      5&sel_instr=%25&begin_hh                                                           
      =0&begin_mi=0&begin_ap=a&end_hh=0&end_mi=0&end_ap=a'''
# split into arguments
bar = sorted(bar.split('&'))
# do the same with the request body that was sent by scrapy
foo =sorted(request.body.split('&'))
# now join these together and find the differences!
zip(foo, bar)
[('begin_ap=a', 'begin_ap=a'),
 ('begin_hh=0', 'begin_hh\n=0'),
 ('begin_mi=0', 'begin_mi=0'),
 ('end_ap=a', 'end_ap=a'),
 ('end_hh=0', 'end_hh=0'),
 ('end_mi=0', 'end_mi=0'),
 ('sel_attr=dummy', 'sel_attr=dummy'),
 ('sel_camp=%25', 'sel_camp=%25'),
 ('sel_camp=dummy', 'sel_camp=dummy'),
 ('sel_crse=', 'sel_crse='),
 ('sel_day=dummy', 'sel_day=dummy'),
 ('sel_from_cred=', 'sel_from_cred='),
 ('sel_insm=%25', 'sel_insm=%25'),
 ('sel_insm=dummy', 'sel_insm=dummy'),
 ('sel_instr=%25', 'sel_instr=%25'),
 ('sel_instr=dummy', 'sel_instr=dummy'),
 ('sel_levl=%25', 'sel_levl=%25'),
 ('sel_levl=dummy', 'sel_levl=dummy\n'),
 ('sel_ptrm=%25', 'sel_ptrm=%25'),
 ('sel_ptrm=dummy', 'sel_ptrm=dummy'),
 ('sel_schd=%25', 'sel_schd\n=%25'),
 ('sel_schd=dummy', 'sel_schd=dummy'),
 ('sel_sess=dummy', 'sel_sess=dummy'),
 ('sel_subj=ART', 'sel_subj=ART'),
 ('sel_title=', 'sel_subj=dummy'),
 ('sel_to_cred=', 'sel_title='),
 ('term_in=201705', 'sel_to_cred=')]

正如你所看到的那样，你错过了＆＃34;虚拟＆＃34;在sel_subj和＆＃39; term_in＆＃39;它不应该存在，但它似乎没有效果：）

使用Scrapy通过html表单的问题

1 个答案: