Django芹菜任务:字典在迭代期间改变了大小

时间:2015-09-26 10:38:31

标签: python django celery django-celery

在Django 1.8中,我有这个视图供用户发布并使用celery函数通知用户的帖子关注者,但它会产生一个相当混乱的错误:

 dictionary changed size during iteration

以下是观点:

  @login_required
    def topic_reply(request, topic_id):
        tform = PostForm()
        topic = Topic.objects.get(pk=topic_id)
        args = {}
        posts = Post.objects.filter(topic= topic)
        posts =  Paginator(posts, DJANGO_SIMPLE_FORUM_REPLIES_PER_PAGE)

        if request.method == 'POST':
            post = PostForm(request.POST)

            if post.is_valid():
                p = post.save(commit = False)
                p.topic = topic
                p.title = post.cleaned_data['title']
                p.body = post.cleaned_data['body']
                p.creator = request.user
                p.user_ip = request.META['REMOTE_ADDR']

                if len(p.title)< 1:
                                p.title=p.body[:60]                                              
                    p.save()

                    #notify followers of the new post creation                        
                    title = 'title' #topic.title
                    link = 'bla' #topic.slug        
                    flwd = request.user
                    flwr_ids = FollowUser.objects.filter(followed=flwd).values('follower_id')
                    flwrs = User.objects.filter(id__in= flwr_ids).values('username','email') 

                    notify_new_post.delay(flwd, flwrs , title, link) #<- here the is the problem

                    return HttpResponseRedirect('/forum/topic/%s/?page=%s'  % (topic.slug, posts.num_pages))
                else:
                    return HttpResponseRedirect('/forum/topic/%s/?page=%s'  % (topic.slug, posts.num_pages))
        else:
            args.update(csrf(request))
            args['form'] = tform
            args['topic'] = topic
            return render_to_response('myforum/reply.html', args, 
                                      context_instance=RequestContext(request))

甚至在传递给函数的任何事情之前都会发生这种情况(我发现celery守护进程没有发生任何事情)

这是芹菜功能:

#@app.task
@task()
def notify_new_post(flwd, flwrs, topic, link):
    print 'post notification \n'
    subject = 'New post'
    from_email = 'noreply@example.com'
    #to_list = [email]     

    for f in flwrs:

        to_email = f['email'].encode('ascii')
        print "[to_email]: " , [to_email]
        args = Context({
            'flwd': flwd,
            'recepient': f['username'],
            'link': link
           })  
       # if to_email !=[]:               
        plaintext = get_template('myforum/email_new_post.txt')
        htmltext = get_template('myforum/email_new_post.html')

        text_content = plaintext.render(args)
        html_content = htmltext.render(args)

        msg = EmailMultiAlternatives(subject, text_content, from_email, [to_email])
        msg.attach_alternative(html_content, "text/html")
        try:
            msg.send()
            print "[to_email]: " , [to_email]
            print 'message sent! \n'
        except Exception as e:  
            print '%s (%s)' % (e.message, type(e))

这对我来说很奇怪,因为“主题”视图的非常相似的任务非常有效。我对此非常感到困惑,所以感谢你的提示。

更新:这是追溯

Environment:


Request Method: POST
Request URL: http://127.0.0.1:8000/forum/reply/52/

Django Version: 1.8.3
Python Version: 2.7.3
Installed Applications:
('django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.humanize',
 'registration',
 'aricle',
 'photo',
 'contact',
 'captcha',
 'pure_pagination',
 'emoticons',
 'debug_toolbar',
 'django_markdown',
 'myforum',
 'userprofile',
 'userpics',
 'djcelery')
Installed Middleware:
(u'debug_toolbar.middleware.DebugToolbarMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'userprofile.middleware.ActiveUserMiddleware')
Traceback:
File "/home/mypc/.projenv/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  132.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/mypc/.projenv/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  22.                 return view_func(request, *args, **kwargs)
File "/home/mypc/myproj/myforum/views.py" in topic_reply
  315.                 notify_new_post.delay(flwd, flwrs , title, link)
File "/home/mypc/.projenv/local/lib/python2.7/site-packages/celery/app/task.py" in delay
  453.         return self.apply_async(args, kwargs)
File "/home/mypc/.projenv/local/lib/python2.7/site-packages/celery/app/task.py" in apply_async
  555.             **dict(self._get_exec_options(), **options)
File "/home/mypc/.projenv/local/lib/python2.7/site-packages/celery/app/base.py" in send_task
  353.                 reply_to=reply_to or self.oid, **options
File "/home/mypc/.projenv/local/lib/python2.7/site-packages/celery/app/amqp.py" in publish_task
  305.             **kwargs
File "/home/mypc/.projenv/local/lib/python2.7/site-packages/kombu/messaging.py" in publish
  161.             compression, headers)
File "/home/mypc/.projenv/local/lib/python2.7/site-packages/kombu/messaging.py" in _prepare
  237.              body) = dumps(body, serializer=serializer)
File "/home/mypc/.projenv/local/lib/python2.7/site-packages/kombu/serialization.py" in dumps
  164.             payload = encoder(data)
File "/usr/lib/python2.7/contextlib.py" in __exit__
  35.                 self.gen.throw(type, value, traceback)
File "/home/mypc/.projenv/local/lib/python2.7/site-packages/kombu/serialization.py" in _reraise_errors
  59.         reraise(wrapper, wrapper(exc), sys.exc_info()[2])
File "/home/mypc/.projenv/local/lib/python2.7/site-packages/kombu/serialization.py" in _reraise_errors
  55.         yield
File "/home/mypc/.projenv/local/lib/python2.7/site-packages/kombu/serialization.py" in dumps
  164.             payload = encoder(data)
File "/home/mypc/.projenv/local/lib/python2.7/site-packages/kombu/serialization.py" in pickle_dumps
  356.         return dumper(obj, protocol=pickle_protocol)

Exception Type: EncodeError at /forum/reply/52/
Exception Value: dictionary changed size during iteration

1 个答案:

答案 0 :(得分:1)

这可能是因为您为flwrs传递的值是django.db.models.query.ValuesQuerySet。 Django查询集被懒惰地评估,我期望它不适合序列化。请记住,您发送到Celery任务并从中返回的所有内容都必须序列化。因此,除了简单的类型或类型之外你不能传递任何其他东西,你知道的事实是可以被序列化的(例如你自己设计的课程或者你内部和外部检查以确保将干净地序列化的课程)

我建议的最低修正是list(flwrs)而不是flwrs。这会将查询集转换为普通list。我还强烈建议将request.user.id传递为flwd而不是用户对象本身。传递ORM对象是获得意外行为的可靠方法。 (Celery文档提到了这一点。)传递id并重新获取Celery任务中的对象是可行的方法。

但是,当我整体查看代码时,我不明白为什么数据库访问是在视图中而不是在Celery任务中执行的。因此,除非我在某个地方使用了线路或变量,否则我会将您的代码更改为仅request.user.id作为flwd,然后在Celery中执行数据库访问任务。所以视图会调用这样的任务:

#notify followers of the new post creation                        
title = 'title' #topic.title
link = 'bla' #topic.slug        
notify_new_post.delay(request.user.id, title, link)

任务就像这样开始:

from django.contrib.auth import get_user_model
@task()
def notify_new_post(flwd_id, topic, link):
    user_model = get_user_model()
    flwd = user_model.objects.get(id=flwd_id)
    flwr_ids = FollowUser.objects.filter(followed=flwd).values('follower_id')
    flwrs = user_model.objects.filter(id__in= flwr_ids).values('username','email') 

(注意最后一行:我假设User是您的Django项目使用的用户模型,因此我使用get_user_model()的返回值(分配到user_model)而不是直接使用User。如果我的假设不正确并且User是其他内容,那么您必须使用User原来的那样。)