不断获取django.db.utils.ProgrammingError:运行4个以上进程时无结果可获取

时间:2019-04-04 13:58:41

标签: python django multiprocessing

我有一个处理数据的脚本。当我用1-3个进程调用它时,它似乎工作正常。但是,一旦尝试使用6个或更多进程运行django错误,就会出现 django.db.utils.ProgrammingError:没有要提取的结果

我已经阅读了几个线程,他们说他们在psychopg2的更高版本中对其进行了修复,因此我尝试升级软件包,当前版本为2.8,但仍然出现错误。

编辑:我注意到的一件事是,当2个进程同时尝试bulk_create时发生错误。我目前正在考虑以某种方式锁定数据库连接,但是我不确定

构成流程的代码

 for profile_name in profiles_to_run:
            t1 = Process(target=self.run_profile_for_inspection, args = (odometer_start,odometer_end,inspection,profile_name,robot_inspection_id))
            t1.start()
            list_of_threads.append(t1)
            thread_count = thread_count + 1


        return list_of_threads

def run_profile_for_inspection(self, odometer_start, odometer_end,portal_inspection, profile_name, robot_inspection_id):
        self.out("Running inspection {} from {}m to {}m for {}".format(portal_inspection.pk, odometer_start, odometer_end,profile_name))
        rerun_profile = ReRunProfile(self._options['db_name'],self.out,profile_name, portal_inspection.pk, robot_inspection_id)
        # rerun_profile.set_odometer_start_and_end(odometer_start,odometer_end)
        rerun_profile.handle_already_created_inspections()

在ReRunProfile中,如果进程太多,则get语句会出错 这是伪代码

   def handle_already_created_inspections(self):
        con = psycopg2.connect(dbname=self.database)
        time.sleep(10)
        robot_inspection_id = self.get_robot_inspection_id(self.portal_inspection_id)
        portal_inspection = Inspection.objects.get(pk=self.portal_inspection_id)
with con.cursor() as cur:
      cur.execute('Select * from data where start < odometer_start and end > odometer_end and profile = profile')
      count = 0
      processed_data = []
      for row in cur:
          processed_data.append(row.process())
          count = count + 1
          if count == 20:
             Profile.objects.bulk_create(processed_data) #errors out here too
             count = 0
             process_data = []



编辑:有人问bulk_create做什么。 基本上,该程序按距离和概要文件拆分数据,然后对其进行处理并将其添加到数据库中。

0 个答案:

没有答案