当尝试将Selenium Webdriver传递给并行执行的功能时,我遇到了错误。 我知道最好在并行运行的函数中初始化Selenium Webdriver,但是,WebDriver的初始化花费的时间太长,而且这些Webdrivers必须用于flask应用程序。目标是在Flask应用启动时准备好网络驱动程序。
class CommetClass:
def __init__(self, urls):
self.drivers = {
1: get_driver(),
2: get_driver(),
3: get_driver()
}
self.urls = urls
@staticmethod
def _task_fn(job_set, driver):
driver.get(url)
...
def task(self):
processes = []
process_id = 0
n_jobs = 3
set_size = (len(self.urls) + n_jobs - 1) // n_jobs
for i in range(n_jobs):
job_set = self.urls[i*set_size:(i+1)*set_size]
if job_set:
process_id += 1
process = multiprocessing.Process(
target=self._task_fn,
args=(job_set, self.drivers.get(process_id))
)
processes.append(process)
process.start()
result = [process.join() for process in processes]
commet = CommetClass(list_of_urls)
# Once the request is made, the following is executed
commet.task()
问题是我收到泡菜错误。