我目前正在尝试使用多重处理来处理psycopg2查询后获得的大结果。
我将结果分成100个列表,然后将其发送以进行多处理。
当我这样做时,出现以下错误
TypeError: can't pickle psycopg2.extensions.connection objects
这是我的psycopg2查询:
def get_employees(self):
logging.info('POSTGRESQL QUERY: get_employees')
try:
self.cur = self.conn.cursor(cursor_factory=RealDictCursor)
self.cur.execute(
"SELECT ..."
)
employees = self.cur.fetchall()
except (Exception, psycopg2.Error) as error:
print("Error while connecting to PostgreSQL", error)
finally:
if self.conn:
return employees
在这种情况下,这是我的多处理方式:
self.db = database.ConnectionClass()
self.employees = self.db.get_employees()
pool = Pool()
for i in range(0, len(self.employees), 100):
pool.apply_async(self.other_function, employees_sublists)
谢谢