Question

我是Python的新手，我刚开始学习Scrapy。

我在蜘蛛文件中的代码如下：

from openbl.items import OpenblItem
import scrapy
import time

class OpenblSpider(scrapy.Spider):
    name='openbl'
    start_url=['http://www.openbl.org/lists/base_1days.txt']
    def parse(self, response):
        #get the content within 'pre', select the 1st element to get the content string.
        #split the space of the content
        content=response.xpath('/pre/text').extract()[0].split()
        # This for loop is used to get the num of element in list content
        # after which the elements of the list are the IPs we desire.
        for i in range(0,len(content)):
            if content[i]=='ip':
                i+=1
                break
            else:
                pass
        # construct a new list content_data for putting IPs in.
        content_data=[]
        # This for loop put useful data(IPs) into the new list above.
        for x in range(i,len(content)):
            content_data.append(content(i))

        for cont in content_data:
            item=OpenblItem()
            item['name']=cont
            item['date']=time.strftime('%Y-%m-%d %H:%M:%S',time.localtime(time.time()))
            item['type']='other'
            yield item

我抓取的网站是：http://www.openbl.org/lists/base_1days.txt 我希望从这个网站获得知识产权[＆＃39; name＆＃39;]

如果有人能回答我的问题，我将不胜感激。

现在出现了一些错误..

 V:\work\openbl>scrapy crawl openbl -o openbl_data.json
2017-01-05 10:46:22 [scrapy.utils.log] INFO: Scrapy 1.3.0 started (bot: openbl)
2017-01-05 10:46:22 [scrapy.utils.log] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'openbl.spiders', 'FEED_URI': 'openbl_data.json', 'SPIDER_MODULES': ['openbl.spiders'], 'BOT_NAME': 'openbl', 'ROBOTSTXT_OBEY': True, 'FEED_FORMAT': 'json'}
2017-01-05 10:46:22 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats']
2017-01-05 10:46:22 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-01-05 10:46:22 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-01-05 10:46:22 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-01-05 10:46:22 [scrapy.core.engine] INFO: Spider opened
2017-01-05 10:46:22 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-01-05 10:46:22 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-01-05 10:46:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.openbl.org/robots.txt> (referer: None)
2017-01-05 10:46:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.openbl.org/lists/base_1days.txt> (referer: None)
2017-01-05 10:46:23 [scrapy.core.scraper] ERROR: Spider error processing <GET http://www.openbl.org/lists/base_1days.txt> (referer: None)
Traceback (most recent call last):
  File "c:\python27\lib\site-packages\scrapy\utils\defer.py", line 102, in iter_errback
    yield next(it)
  File "c:\python27\lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 29, in process_spider_output
    for x in result:
  File "c:\python27\lib\site-packages\scrapy\spidermiddlewares\referer.py", line 22, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "c:\python27\lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "c:\python27\lib\site-packages\scrapy\spidermiddlewares\depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "V:\work\openbl\openbl\spiders\openbl_spider.py", line 11, in parse
    content=response.xpath('/pre/text').extract()[0].split()
IndexError: list index out of range
2017-01-05 10:46:23 [scrapy.core.engine] INFO: Closing spider (finished)
2017-01-05 10:46:23 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 454,
 'downloader/request_count': 2,
 'downloader/request_method_count/GET': 2,
 'downloader/response_bytes': 4907,
 'downloader/response_count': 2,
 'downloader/response_status_count/200': 2,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2017, 1, 5, 2, 46, 23, 515000),
 'log_count/DEBUG': 3,
 'log_count/ERROR': 1,
 'log_count/INFO': 7,
 'response_received_count': 2,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'spider_exceptions/IndexError': 1,
 'start_time': datetime.datetime(2017, 1, 5, 2, 46, 22, 383000)}
2017-01-05 10:46:23 [scrapy.core.engine] INFO: Spider closed (finished)

Answer 1

此代码存在几个问题，首先是

start_urls否则我认为它不会爬行任何东西

您收到该错误是因为它是纯文本文件，response.body实际上是纯文本。没有标签。所以你得到一个索引超出范围的异常。您可以简单地将其作为纯文本处理，并通过正则表达式提取信息，按\n分割等等。有太多方法可以做到这一点。

另外，不要使用循环变量，在我喜欢的情况下。这感觉不对劲。如果要查找列表中第一次出现的索引，则有index()函数。要么 What is the best way to get the first item from an iterable matching a condition?

当我使用Scrapy进行爬网时，不会出现任何错误，但不会抓取任何内容

1 个答案: