Question

我对每个插入查询（小查询）都有一个问题，它在异步中在芹菜任务中执行。在同步模式下，当我插入所有做得很好，但当它在apply_async（）执行时我得到这个：

OperationTimedOut('errors=errors=errors={}, last_host=***.***.*.***, last_host=None, last_host=None',)

回溯：

Traceback (most recent call last):
  File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
    return self.run(*args, **kwargs)
  File "/var/nfs_www/***/www_v1/app/mods/news_feed/tasks.py", line 26, in send_new_comment_reply_notifications
    send_new_comment_reply_notifications_method(comment_id)
  File "/var/nfs_www/***www_v1/app/mods/news_feed/methods.py", line 83, in send_new_comment_reply_notifications
    comment_type='comment_reply'
  File "/var/nfs_www/***/www_v1/app/mods/news_feed/models/storage.py", line 129, in add
    CommentsFeed(**kwargs).save()
  File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/cqlengine/models.py", line 531, in save
    consistency=self.__consistency__).save()
  File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/cqlengine/query.py", line 907, in save
    self._execute(insert)
  File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/cqlengine/query.py", line 786, in _execute
    tmp = execute(q, consistency_level=self._consistency)
  File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/cqlengine/connection.py", line 95, in execute
    result = session.execute(query, params)
  File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/cassandra/cluster.py", line 1103, in execute
    result = future.result(timeout)
  File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/cassandra/cluster.py", line 2475, in result
    raise OperationTimedOut(errors=self._errors, last_host=self._current_host)
OperationTimedOut: errors={}, last_host=***.***.*.***

有没有人有关于问题的想法？

我发现了这个When cassandra-driver was executing the query, cassandra-driver returned error OperationTimedOut，但我的查询很少，只有芹菜任务才有问题。

更新

我做了一个测试任务，它也引发了这个错误。

@celery.task()
def test_task_with_cassandra():
    from app import cassandra_session
    cassandra_session.execute('use news_feed')
    return 'Done'

更新2： 做到了这个：

@celery.task()
def test_task_with_cassandra():
    from cqlengine import connection
    connection.setup(app.config['CASSANDRA_SERVERS'], port=app.config['CASSANDRA_PORT'],
                     default_keyspace='test_keyspace')
    from .models import Feed
    Feed.objects.count()
    return 'Done'

得到了这个：

NoHostAvailable('Unable to connect to any servers', {'***.***.*.***': OperationTimedOut('errors=errors=Timed out creating connection, last_host=None, last_host=None',)})

从shell我可以连接到它

更新3 ：从github问题上删除的线程（在我的电子邮件中找到）:(这对我也有用）实际上，我将CQLengine插入Celery：

from celery import Celery
from celery.signals import worker_process_init, beat_init
from cqlengine import connection
from cqlengine.connection import (
    cluster as cql_cluster, session as cql_session)

def cassandra_init():
    """ Initialize a clean Cassandra connection. """
    if cql_cluster is not None:
        cql_cluster.shutdown()
    if cql_session is not None:
        cql_session.shutdown()
    connection.setup()

# Initialize worker context for both standard and periodic tasks.
worker_process_init.connect(cassandra_init)
beat_init.connect(cassandra_init)

app = Celery()

这很粗糙，但有效。我们应该在常见问题解答中添加此代码段吗？

Answer 1

我有类似的问题。这似乎与在任务之间共享Cassandra会话有关。我通过为每个线程创建一个会话来解决它。确保从任务中调用get_session()然后执行此操作：

thread_local = threading.local()

def get_session():
    if hasattr(thread_local, "cassandra_session"):
        return thread_local.cassandra_session

    cluster = Cluster(settings.CASSANDRA_HOSTS)
    session = cluster.connect(settings.CASSANDRA_KEYSPACE)

    thread_local.cassandra_session = session

    return session

Answer 2

受Ron的回答启发，我想出了以下代码来放入tasks.py：

import threading
from django.conf import settings
from cassandra.cluster import Cluster
from celery.signals import worker_process_init,worker_process_shutdown

thread_local = threading.local()

@worker_process_init.connect
def open_cassandra_session(*args, **kwargs):
    cluster = Cluster([settings.DATABASES["cassandra"]["HOST"],], protocol_version=3)
    session = cluster.connect(settings.DATABASES["cassandra"]["NAME"])
    thread_local.cassandra_session = session

@worker_process_shutdown.connect
def close_cassandra_session(*args,**kwargs):
    session = thread_local.cassandra_session
    session.shutdown()
    thread_local.cassandra_session = None

当芹菜工人流程开始和停止时，这个简洁的解决方案将自动打开/关闭cassandra会话。

附注：protocol_version = 3，因为Cassandra 2.1仅支持协议版本3及更低版本。

Answer 3

其他答案对我没有用，但问题是更新3＆＃39;没有。这就是我最终得到的结果（问题中建议的小更新）：

from celery.signals import worker_process_init
from cassandra.cqlengine import connection
from cassandra.cqlengine.connection import (
    cluster as cql_cluster, session as cql_session)

def cassandra_init(*args, **kwargs):
    """ Initialize a clean Cassandra connection. """
    if cql_cluster is not None:
        cql_cluster.shutdown()
    if cql_session is not None:
        cql_session.shutdown()
    connection.setup(settings.DATABASES["cassandra"]["HOST"].split(','), settings.DATABASES["cassandra"]["NAME"])

# Initialize worker context (only standard tasks)
worker_process_init.connect(cassandra_init)

Answer 4

使用django-cassandra-engine，以下代码可以为我解决问题：

db_connection = connections['cassandra']

@worker_process_init.connect
def connect_db(**_):
    db_connection.reconnect()


@worker_shutdown.connect
def disconnect(**_):
    db_connection.connection.close_all()

看看here

对于Celery任务中的每个查询，Python cassandra-driver OperationTimeOut

4 个答案: