Django - 复杂的查询

时间:2012-11-05 18:51:43

标签: python django

假设我有两个模型:

class Profile(models.Model):
    #some fields here

class Ratings(models.Model):
    profile = models.ForeignKey(profile)
    category = models.IntegerField()
    points = models.IntegerField()

假设遵循MySQL表“评级”:

profile    |    category    |    points
   1                1               10
   1                1               4
   1                2               10
   1                3               0
   1                4               10
   1                4               10
   1                4               10
   1                5               0

我的POST数据中包含以下值以及其他字段值:

category_1_avg_val = 7
category_2_avg_val = 5
category_3_avg_val = 5
category_4_avg_val = 7
category_5_avg_val = 9

我想过滤那些为高于或等于所需值的类别计算平均评分的个人资料。

有些过滤器最初应用为:

q1 = [('associated_with', search_for),
      ('profile_type__slug__exact', profile_type),
      ('gender__in', gender),
      ('rank__in', rank),
      ('styles__style__in', styles),
      ('age__gte', age_from),
      ('age__lte', age_to)]
q1_list = [Q(x) for x in q1 if x[1]]

q2 = [('user__first_name__icontains', search_term),
      ('user__last_name__icontains', search_term),
      ('profile_type__name__icontains', search_term),
      ('styles__style__icontains', search_term),
      ('rank__icontains', search_term)]
q2_list = [Q(x) for x in q2 if x[1]]

if q1_list:
    objects = Profile.objects.filter(
        reduce(operator.and_, q1_list))

if q2_list:
    if objects:
        objects = objects.filter(
            reduce(operator.or_, q2_list))
    else:
        objects = Profile.objects.filter(
            reduce(operator.or_, q2_list))

if order_by_ranking_level == 'desc':
    objects = objects.order_by('-ranking_level').distinct()
else:
    objects = objects.order_by('ranking_level').distinct()

现在我要过滤其(平均分数)(按类别分组)> =(后期类别的平均值)

的个人资料

我试图逐一做到这一点

objects = objects.filter(
    ratings__category=1) \
    .annotate(avg_points=Avg('ratings__points'))\
    .filter(avg_points__gte=category_1_avg_val)


objects = objects.filter(
    ratings__category=2) \
    .annotate(avg_points=Avg('ratings__points'))\
    .filter(avg_points__gte=category_2_avg_val)

但我认为这是错误的。请帮帮我。如果return是一个很棒的查询集。

被修改 使用hynekcer发布的答案,我提出了稍微不同的解决方案,因为我已经查询了需要根据评级过滤的配置文件。

def check_ratings_avg(pr, rtd):
    ok = True
    qr = Ratings.objects.filter(profile__id=pr.id) \
        .values('category')\
        .annotate(points_avg=Avg('points'))
    qr = {i['category']:i['points_avg'] for i in qr}

    for cat in rtd:
        val = rtd[cat]
        if qr[cat] >= val:
            pass
        else:
            ok = False
            break
    return ok


rtd = {1: category_1_avg_val, 2: category_2_avg_val, 3: category_3_avg_val,
       4: category_4_avg_val, 5: category_5_avg_val}
objects = [i for i in objects if check_ratings_avg(i, rtd)]

2 个答案:

答案 0 :(得分:6)

您的复杂查询需要原则中的子查询。可能的解决方案是:

  • 'extra' queryset methodraw SQL查询撰写的子查询。它不是DRY,并且一些数据库后端不支持它,例如某些版本的MySQL,但是从Django 1.1开始,子查询的使用方式有限。
  • 将中间结果保存到数据库中的临时表中。这在Django中并不好。
  • 在Python中通过循环模拟外部查询。最好的通用解决方案。 Python中的第一个查询聚合的数据库数据循环可以足够快地聚合和过滤数据。

A)由Python模拟的子查询

from django.db.models import Q, Avg
from itertools import groupby
from myapp.models import Profile, Ratings

def iterator_filtered_by_average(dictionary):
    qr = Ratings.objects.values('profile', 'category', 'points').order_by(
            'profile', 'category').annotate(points_avg=Avg('points'))
    f = Q()
    for k, v in dictionary.iteritems():
        f |= Q(category=k, points_avg__gte=v)
    for profile, grp in groupby(qr.filter(f).values('profile')):
        if len(list(grp)) == len(dictionary):
            yield profile

#example
FILTER_DATA = {1:category_1_avg_val, 2:category_2_avg_val, 3:category_3_avg_val,
               4:category_4_avg_val, 5:category_5_avg_val}
for row in iterator_filtered_by_average(FILTER_DATA):
    print row

这是原始问题的简单解决方案,没有后来的额外要求。

B)子查询解决方案
有必要提供更详细的问题版本,因为如果初始过滤器基于类型ManyToManyField的某个字段,并且还包含distinct子句:

# objects:  QuerySet that you get from your initial filters. Not yet executed.
if rtd:
    # Method `as_nested_sql` removes the `order_by` clase, unlike `as_sql`
    subquery3 = objects.values('id').query \
            .get_compiler(connection=connection).as_nested_sql()
    subquery2 = ("""SELECT profile_id, category, avg(points) AS points_avg
          FROM myapp_ratings
          WHERE profile_id in
          ( %s
          ) GROUP BY profile_id, category
            """ % subquery3[0], subquery3[1]
    )
    where_sql = ' OR '.join(
            'category = %d AND points_avg >= %%s' % cat for cat in rtd.keys()
    )
    subquery = (
        """SELECT profile_id
        FROM
        ( %s
        ) subquery2
        WHERE %s
        GROUP BY profile_id
        HAVING count(*) = %s
        """ % (subquery2[0], where_sql, len(rtd)),
        subquery2[1] + tuple(rtd.values())
    )
    assert order_by_ranking_level in ('asc', 'desc')
    mainquery = ("""SELECT myapp_profile.* FROM myapp_profile
      INNER JOIN
      ( %s
      ) subquery ON subquery.profile_id=myapp_profile.id
      ORDER BY ranking_level %s"""
        % (subquery[0], order_by_ranking_level), subquery[1]
    )
    objects = Profile.objects.raw(mainquery[0], params=mainquery[1])
return objects

将所有字符串myapp替换为name_of_your_application

此代码生成的SQL示例

SELECT myapp_profile.* FROM myapp_profile
  INNER JOIN
  ( SELECT profile_id
    FROM
    ( SELECT profile_id, category, avg(points) AS points_avg
      FROM myapp_ratings
      WHERE profile_id IN
      ( SELECT U0.`id` FROM `myapp_profile` U0 WHERE U0.`ranking_level` >= 4
      ) GROUP BY profile_id, category
    ) subquery2
    WHERE category = 1 AND points_avg >= 7 OR category = 2 AND points_avg >= 5
       OR category = 3 AND points_avg >= 5 OR category = 4 AND points_avg >= 7
       OR category = 5 AND points_avg >= 9
    GROUP BY profile_id
    HAVING count(*) = 5
  ) subquery ON subquery.profile_id=myapp_profile.id
  ORDER BY ranking_level asc

(这个SQL是为了更好的可读性,手动解析字符串%s替换为参数,但是出于安全原因,数据库引擎会接收未解析的参数。)


您的问题是由于Django生成的子查询支持不多。只有更复杂查询文档中的示例才会创建子查询。 (例如aggregate之后的annotatecount之后的annotateaggregate之后的distinctannotate之后的distinct }或annotate之后)复杂的嵌套聚合被简化为一个意外的查询。

不鼓励对第一个查询过滤的每个对象执行新的单独SQL查询的所有其他解决方案进行生产,尽管它们对于测试任何更好的解决方案的结果非常有用。

答案 1 :(得分:0)

您可以向经理添加方法

# Untested code
class ProfileManager(models.Manager):
    def with_category_average(self, cat, avg):
        # Give each filter a unique annotation key
        key = 'avg_pts_' + str(cat)
        return self.filter(ratings__category=cat) \
                   .annotate(**{key: Avg('ratings__points')}) \
                   .filter(**{key + '__gte': avg})

    # Expects a dict of `cat: avg` pairs
    def filter_by_averages(self, avg_dict):
        qs = self.get_query_set()
        for key, val in avg_dict.items():
            qs &= self.with_category_average(key, val)
        return qs