如何在scrapy-redis-bloomfilter中设置?

时间:2018-07-23 04:05:51

标签: redis scrapy bloom-filter

谢谢男人。 此组件,scrapy-redis-bloomfilter,https://github.com/Python3WebSpider/ScrapyRedisBloomFilter

我想知道为什么当我重复使用它时会使用这么多的内存吗?

MYSQL_HOST='127.0.0.1'
MYSQL_DBNAME='abc'
MYSQL_USER='root'
MYSQL_PASSWORD='abc'

SCHEDULER = "scrapy_redis_bloomfilter.scheduler.Scheduler"
DUPEFILTER_CLASS = "scrapy_redis_bloomfilter.dupefilter.RFPDupeFilter"
BLOOMFILTER_HASH_NUMBER = 6
BLOOMFILTER_BIT = 30
SCHEDULER_PERSIST = True
REDIS_PARAMS  = {'password':'abcdefg','db':1}

urls:37000,mem:580m

swap-mem

redis-Manager

0 个答案:

没有答案