不能同时运行多个蜘蛛

时间:2020-05-18 18:52:42

标签: python multithreading selenium scrapy

我正在做一个沙皮项目,我想同时运行多个蜘蛛这是从脚本运行蜘蛛的代码

from spiders.product_spider import ProductSpider
from spiders.another_spider import AnotherSpider
from scrapy.crawler import CrawlerProcess
from scrapy.settings import Settings
from butik.butik import settings as my_settings


crawler_settings = Settings()
crawler_settings.setmodule(my_settings)
process = CrawlerProcess(settings=crawler_settings)

process.crawl(ProductSpider)
process.crawl(AnotherSpider)
process.crawl(AnotherSpider)
process.crawl(ProductSpider)

process.start() 

但是问题是运行文件时需要花费一定的时间(如果我运行两个蜘蛛需要50秒,如果运行4个蜘蛛需要1:40分钟)

注意:我正在从硒中下载页面

0 个答案:

没有答案