Question

我正在尝试根据每个公司设置一个多处理池。其中，对于每个公司，并行启动一个线程池。

我已经使用pathos.multiprocessing创建了一个多处理池，在其中调用了一个单独的函数来尝试启动线程池。在这里，ThrdPool是一个自定义类，可以在主类文件夹位置本地使用，也可以导入到主类中。当我尝试在各自的函数中调用此类时，出现“未定义Thrdpool”错误。

我尝试了所有类型的导入，例如导入ThrdPool 从ThrdPool导入ThrdPool 从ThrdPool导入* 我什至将文件从ThreadPool重命名为ThrdPool，但还是没有运气。我可以通过添加main方法来运行ThrdPool.py，而不会遇到任何麻烦。

from pathos import multiprocessing as multiprocessing
from functools import partial
from ThrdPool import *

class APICall():
     def __init__(self):
     self.host = config.get('splunk_search_section', 'host_name')
     self.port = config.get('splunk_search_section', 'host_port')
     self.username = config.get('splunk_search_section', 'user_name')
     self.password = config.get('splunk_search_section', 'password')
 def process(self):
     company_list = self.fetch_onboarded_companies_from_customer_csv()
     saved_search_list_file = os.path.join(code_dir_path, "resources\\saved_search_template.txt")
     try:
         with open(saved_search_list_file, "r") as ss_file_pointer:
             saved_search_list = ss_file_pointer.readlines()
     except IOError as ie:
         self.logging.error(f"Error occurred while accessing the Saved Search file reason being , {ie}")
         raise IOError("IO Error occurred while accessing the Saved Search file.")
     finally:
         ss_file_pointer.close()
     # Creating a process pool for each company key, to increase the throughput.
     # Assuming 4 processors to be max optimistically.
     p = multiprocessing.Pool(processes=4)
     # for each_cpy in company_list:
     list_length_company = len(company_list)
     array_of_numbers = [x for x in range(0, list_length_company)]
     ssl = saved_search_list
     cl = company_list
     func = partial(self.processing_saved_search_per_company, ssl, cl)
     p.map(func, array_of_numbers)
     p.close()
     p.join()
 def processing_saved_search_per_company(self, saved_search_list, company_list, each_cpy_index):
     company_key = company_list[each_cpy_index]
     print("Company Key : " + company_key)
     self.logging.info(f"processing the saved search for company {company_key}")
     each_cpy = company_list[each_cpy_index]
     array_of_numbers = [x for x in range(0, len(saved_search_list))]
     # Creating a Thread pool of 5 threads to optimistically increase the throughput of saved search processing.
     thread_pool = ThrdPool(5)
     function1 = partial(self.run_saved_search_per_company, saved_search_list, each_cpy)
     thread_pool.map(function1, array_of_numbers)
     thread_pool.wait_completion()

正在等待要创建的线程池，但是线程池的导入失败。即使我注释了线程池，partial的下一行也失败了，并出现以下错误。

回溯（最近通话最近）：文件“ C：/Users/sp/src/APICall.py”，行176，在 APICall（）。process（）正在处理文件“ C：/Users/sp/APICall.py”，第92行公司密钥：105261 p.map（func，array_of_numbers）地图中的文件“ C：\ Users \ xxxxx \ AppData \ Roaming \ Python \ Python37 \ site-> packages \ multiprocess \ pool.py”，行268 返回self._map_async（func，可迭代，mapstar，chunksize）.get（）在获取文件“ C：\ Users \ xxxxxx \ AppData \ Roaming \ Python \ Python37 \ site-packages \ multiprocess \ pool.py”的行657中提高自我价值 NameError：名称“ ThrdPool”未定义

在这方面寻求帮助，是否有可能在多进程池下旋转多个线程？

谢谢沙希德

无法产生具有多进程池的线程池

0 个答案: