使用Postgres优化Django queryset查询

时间:2019-02-27 11:23:15

标签: python django postgresql django-queryset

我正在使用Django 2.1和python 3.6 我有一个包含其他查询的查询,当要处理大量数据时,评估时间似乎太长。 基本上它达到了db用户超时(10秒),因此查询失败,并且当我尝试不超时时,将近一分钟后我从DB(postgres)收到以下错误消息:

ERROR:  canceling statement due to conflict with recovery
DETAIL:  User query might have needed to see row versions that must be removed.

我的问题是-如何使查询更高效? (所有使用的字段都在数据库中建立了索引)

这是查询:

migration_set__qs = Migration.objects.filter(
        migration_id=migration_id,
        migration_version=migration_version,
        migration_data__generated_id__isnull=False
    ).values_list(
        'object_id',
        flat=True
    )

containers__qs = Container.objects.all().exclude(
            Q(id__in=migration_set__qs) | Q(created_at__gte=turned_on_date)
        )

limited_containers = containers__qs[0:10]
num_containers_processed += limited_containers.count()

'count()'触发对查询的评估,并在那里中断。

1 个答案:

答案 0 :(得分:0)

我要尝试的第一件事是将migration_set_qs更改为Subquery

migration_set__qs = Migration.objects.filter(
    migration_id=migration_id,
    migration_version=migration_version,
    migration_data__generated_id__isnull=False
).values(
    'object_id',
)

containers__qs = Container.objects.all().exclude(
    Q(id__in=Subquery(migration_set__qs)) | Q(created_at__gte=turned_on_date)
)

limited_containers = containers__qs[0:10]
num_containers_processed += limited_containers.count()

尽管我不确定它能否解决您的所有问题。如果有任何不同,请告诉我,如果我有其他任何想法,我也会更新此答案。