我正在做一个沙皮项目,我想同时运行多个蜘蛛这是从脚本运行蜘蛛的代码
from spiders.product_spider import ProductSpider
from spiders.another_spider import AnotherSpider
from scrapy.crawler import CrawlerProcess
from scrapy.settings import Settings
from butik.butik import settings as my_settings
crawler_settings = Settings()
crawler_settings.setmodule(my_settings)
process = CrawlerProcess(settings=crawler_settings)
process.crawl(ProductSpider)
process.crawl(AnotherSpider)
process.crawl(AnotherSpider)
process.crawl(ProductSpider)
process.start()
但是问题是运行文件时需要花费一定的时间(如果我运行两个蜘蛛需要50秒,如果运行4个蜘蛛需要1:40分钟)
注意:我正在从硒中下载页面