谢谢男人。 此组件,scrapy-redis-bloomfilter,https://github.com/Python3WebSpider/ScrapyRedisBloomFilter
我想知道为什么当我重复使用它时会使用这么多的内存吗?
MYSQL_HOST='127.0.0.1'
MYSQL_DBNAME='abc'
MYSQL_USER='root'
MYSQL_PASSWORD='abc'
SCHEDULER = "scrapy_redis_bloomfilter.scheduler.Scheduler"
DUPEFILTER_CLASS = "scrapy_redis_bloomfilter.dupefilter.RFPDupeFilter"
BLOOMFILTER_HASH_NUMBER = 6
BLOOMFILTER_BIT = 30
SCHEDULER_PERSIST = True
REDIS_PARAMS = {'password':'abcdefg','db':1}
urls:37000,mem:580m