我有一个处理数据的脚本。当我用1-3个进程调用它时,它似乎工作正常。但是,一旦尝试使用6个或更多进程运行django错误,就会出现 django.db.utils.ProgrammingError:没有要提取的结果
我已经阅读了几个线程,他们说他们在psychopg2的更高版本中对其进行了修复,因此我尝试升级软件包,当前版本为2.8,但仍然出现错误。
编辑:我注意到的一件事是,当2个进程同时尝试bulk_create
时发生错误。我目前正在考虑以某种方式锁定数据库连接,但是我不确定
构成流程的代码
for profile_name in profiles_to_run:
t1 = Process(target=self.run_profile_for_inspection, args = (odometer_start,odometer_end,inspection,profile_name,robot_inspection_id))
t1.start()
list_of_threads.append(t1)
thread_count = thread_count + 1
return list_of_threads
def run_profile_for_inspection(self, odometer_start, odometer_end,portal_inspection, profile_name, robot_inspection_id):
self.out("Running inspection {} from {}m to {}m for {}".format(portal_inspection.pk, odometer_start, odometer_end,profile_name))
rerun_profile = ReRunProfile(self._options['db_name'],self.out,profile_name, portal_inspection.pk, robot_inspection_id)
# rerun_profile.set_odometer_start_and_end(odometer_start,odometer_end)
rerun_profile.handle_already_created_inspections()
在ReRunProfile中,如果进程太多,则get语句会出错 这是伪代码
def handle_already_created_inspections(self):
con = psycopg2.connect(dbname=self.database)
time.sleep(10)
robot_inspection_id = self.get_robot_inspection_id(self.portal_inspection_id)
portal_inspection = Inspection.objects.get(pk=self.portal_inspection_id)
with con.cursor() as cur:
cur.execute('Select * from data where start < odometer_start and end > odometer_end and profile = profile')
count = 0
processed_data = []
for row in cur:
processed_data.append(row.process())
count = count + 1
if count == 20:
Profile.objects.bulk_create(processed_data) #errors out here too
count = 0
process_data = []
编辑:有人问bulk_create做什么。 基本上,该程序按距离和概要文件拆分数据,然后对其进行处理并将其添加到数据库中。