将硒驱动程序传递给并行任务

时间:2020-07-28 08:05:25

标签: python-3.x selenium-webdriver flask parallel-processing

当尝试将Selenium Webdriver传递给并行执行的功能时,我遇到了错误。 我知道最好在并行运行的函数中初始化Selenium Webdriver,但是,WebDriver的初始化花费的时间太长,而且这些Webdrivers必须用于flask应用程序。目标是在Flask应用启动时准备好网络驱动程序。

class CommetClass:
    def __init__(self, urls):
        self.drivers = {
            1: get_driver(),
            2: get_driver(),
            3: get_driver()
        }
        self.urls = urls

    @staticmethod
    def _task_fn(job_set, driver):
        driver.get(url)
        ...

    def task(self):
        processes = []
        process_id = 0

        n_jobs = 3
        set_size = (len(self.urls) + n_jobs - 1) // n_jobs
        for i in range(n_jobs):
            job_set = self.urls[i*set_size:(i+1)*set_size]
            if job_set:
                process_id += 1
                process = multiprocessing.Process(
                    target=self._task_fn,
                    args=(job_set, self.drivers.get(process_id))
                )
                processes.append(process)
                process.start()
        result = [process.join() for process in processes]


commet = CommetClass(list_of_urls)
# Once the request is made, the following is executed
commet.task()

问题是我收到泡菜错误。

0 个答案:

没有答案