在Web浏览器中访问页面后，响应缓慢变化

时间：2018-06-21 17:10:31

标签： python scrapy rss

下面是我为爬行RSS提要并提取第一个链接和图像标题并将其保存到文本文件而编写的蜘蛛：

import scrapy

class artSpider(scrapy.Spider):
    name = "metart"

    start_urls = ['https://www.metmuseum.org/art/artwork-of-the-day?rss=1']

    def parse(self, response):
        title = response.xpath('//item/title/text()').extract_first().replace(" ", "_")
        description = response.xpath('//item/description/text()').extract_first()

        lnkstart = description.find("https://image")
        lnkcut1 = description.find("web-highlight")
        lnkcut2 = lnkcut1 + 13
        lnkend = description.find(".jpg") + 4

        link = description[lnkstart:lnkcut1] + "original" + description[lnkcut2:lnkend]

        ttlstart = description.find("who=") + 4
        ttlend = description.find("&rss=1")

        filename = "/path/to/save/folder/" + description[ttlstart:ttlend].replace("+", "_") + "-" + title + ".jpg"

        print(filename)
        print(link)

        filename_file = open('filename_metart.txt', 'w')
        filename_file.write(filename)
        filename_file.close
        link_file = open('link_metart.txt', 'w')
        link_file.write(link)
        link_file.close

它经过大都会博物馆的“每日艺术品” RSS feed，并找到最新的艺术品。然后解析标题和原始链接（RSS提要中的链接用于缩略图）并将其保存到单独的文本文件中。下载图像时，标题用于生成文件名。

我知道解析函数是一团糟，并且保存链接和文件名的方式很丑陋。我只是从Scrapy开始，只是想让Spider工作，因为它只是我正在从事的更广泛项目的一部分。

该项目本身从当天的提要和网站的艺术品/图像/照片中提取图片，下载图像并将其设置为桌面背景幻灯片。

我的问题是，当我关闭蜘蛛时，它会返回“ NoneType”响应。但是，如果我在浏览器（https://metmuseum.org/art/artwork-of-the-day?rss=1中转到RSS提要，然后运行蜘蛛，它将正常工作。

进程失败

  1. 致电scrapy crawl metart
    • 输出显示变量title的类型为NoneType
  2. 重复步骤1会得到相同的输出

有效的过程

  1. 在网络浏览器中打开https://metmuseum.org/art/artwork-of-the-day?rss=1
  2. 致电scrapy crawl metart
    • 成功将链接和文件名保存到相关的文本文件中

我已经做过一些故障排除

我使用了Scrapy shell来精确地复制蜘蛛经历的过程，并且效果很好。我从头开始经历了蜘蛛所经历的每一步：

fetch("https://metmuseum.org/art/artwork-of-the-day?rss=1")

然后在parse函数的每一行中键入。这样可以很好地工作，并且可以保存正确的链接和文件名，而无需先在Web浏览器中打开URL。

为完整起见，以下是我的Scrapy项目settings.py文件：

BOT_NAME = 'wallpaper_scraper'

SPIDER_MODULES = ['wallpaper_scraper.spiders']
NEWSPIDER_MODULE = 'wallpaper_scraper.spiders'

# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'cchowgule for a wallpaper slideshow'

# Obey robots.txt rules
ROBOTSTXT_OBEY = True

对于在网络浏览器中打开URL的行为可能如何影响蜘蛛的响应，我感到非常困惑。

清理我的解析函数的任何帮助也很可爱。

谢谢

更新

在0535 UTC，我运行scrapy crawl metart并得到以下响应：

2018-06-23 11:02:54 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: wallpaper_scraper)
2018-06-23 11:02:54 [scrapy.utils.log] INFO: Versions: lxml 4.2.1.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.4.0, w3lib 1.19.0, Twisted 17.9.0, Python 2.7.15 (default, May  1 2018, 05:55:50) - [GCC 7.3.0], pyOpenSSL 17.5.0 (OpenSSL 1.1.0h  27 Mar 2018), cryptography 2.2.2, Platform Linux-4.16.0-2-amd64-x86_64-with-debian-buster-sid
2018-06-23 11:02:54 [scrapy.crawler] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'wallpaper_scraper.spiders', 'SPIDER_MODULES': ['wallpaper_scraper.spiders'], 'ROBOTSTXT_OBEY': True, 'USER_AGENT': 'cchowgule for a wallpaper slideshow', 'BOT_NAME': 'wallpaper_scraper'}
2018-06-23 11:02:54 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats']
2018-06-23 11:02:54 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-06-23 11:02:54 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-06-23 11:02:54 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2018-06-23 11:02:54 [scrapy.core.engine] INFO: Spider opened
2018-06-23 11:02:54 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-06-23 11:02:54 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-06-23 11:02:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.metmuseum.org/robots.txt> (referer: None)
2018-06-23 11:02:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.metmuseum.org/art/artwork-of-the-day?rss=1> (referer: None)
2018-06-23 11:02:56 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.metmuseum.org/art/artwork-of-the-day?rss=1> (referer: None)
Traceback (most recent call last):
  File "/home/cchowgule/.local/lib/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/cchowgule/WD/pyenvscrapy/wallpaper_scraper/wallpaper_scraper/spiders/metart.py", line 9, in parse
    title = response.xpath('//item/title/text()').extract_first().replace(" ", "_")
AttributeError: 'NoneType' object has no attribute 'replace'
2018-06-23 11:02:57 [scrapy.core.engine] INFO: Closing spider (finished)
2018-06-23 11:02:57 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 646,
 'downloader/request_count': 2,
 'downloader/request_method_count/GET': 2,
 'downloader/response_bytes': 1725,
 'downloader/response_count': 2,
 'downloader/response_status_count/200': 2,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2018, 6, 23, 5, 32, 57, 65104),
 'log_count/DEBUG': 3,
 'log_count/ERROR': 1,
 'log_count/INFO': 7,
 'memusage/max': 52793344,
 'memusage/startup': 52793344,
 'response_received_count': 2,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'spider_exceptions/AttributeError': 1,
 'start_time': datetime.datetime(2018, 6, 23, 5, 32, 54, 464466)}
2018-06-23 11:02:57 [scrapy.core.engine] INFO: Spider closed (finished)

我跑了scrapy crawl metart 5次，结果相同。

然后，我打开浏览器，转到https://metmuseum.org/art/artwork-of-the-day?rss=1，然后再次运行scrapy crawl metart。这次它正常工作了……我不明白。

0 个答案:

没有答案
相关问题
最新问题