Question

如果Cassandra数据库中存在或不存在当前数据ID，那么这是我的查询

row = session.execute("SELECT * FROM articles where id = %s", [id])

Kafka中已解析的消息，然后确定cassandra数据库中是否存在此消息（如果该消息不存在），则应执行插入操作，如果确实存在，则不应将其插入数据中。

messages = consumer.get_messages(count=25)

    if len(messages) == 0:
        print 'IDLE'
        sleep(1)
        continue

    for message in messages:
        try:
            message = json.loads(message.message.value)
            data = message['data']
            if data:
                for article in data:
                    source = article['source']
                    id = article['id']
                    title = article['title']
                    thumbnail = article['thumbnail']
                    #url = article['url']
                    text = article['text']
                    print article['created_at'],type(article['created_at'])
                    created_at = parse(article['created_at'])
                    last_crawled = article['last_crawled']
                    channel = article['channel']#userid
                    category = article['category']
                    #scheduled_for = created_at.replace(minute=created_at.minute + 5, second=0, microsecond=0)
                    scheduled_for=(datetime.utcnow() + timedelta(minutes=5)).replace(second=0, microsecond=0)
                    row = session.execute("SELECT * FROM articles where id = %s", [id])
                    if len(list(row))==0:
                    #id parse base62
                        ids = [id[0:2],id[2:9],id[9:16]]
                        idstr=''
                        for argv in ids:
                            num = int(argv)
                            idstr=idstr+encode(num)
                        url='http://weibo.com/%s/%s?type=comment' % (channel,idstr)
                        session.execute("INSERT INTO articles(source, id, title,thumbnail, url, text, created_at, last_crawled,channel,category) VALUES (%s,%s, %s, %s, %s, %s, %s, %s, %s, %s)", (source, id, title,thumbnail, url, text, created_at, scheduled_for,channel,category))
                        session.execute("INSERT INTO schedules(source,type,scheduled_for,id) VALUES (%s, %s, %s,%s) USING TTL 86400", (source,'article', scheduled_for, id))
                        log.info('%s %s %s %s %s %s %s %s %s %s' % (source, id, title,thumbnail, url, text, created_at, scheduled_for,channel,category))


        except Exception, e:
            log.exception(e)
            #log.info('error %s %s' % (message['url'],body))
            print e
            continue

修改

我有一个ID只有一个唯一的表行，我希望这样。一旦我为唯一ID添加了不同的scheduled_for时间，我的系统就会崩溃。如果len（list（row））== 0：添加这个是正确的想法，但之后我的系统非常慢。

这是我的表格描述：

DROP TABLE IF EXISTS schedules;

CREATE TABLE schedules (
 source text,
 type text,
 scheduled_for timestamp,
 id text,
 PRIMARY KEY (source, type, scheduled_for, id)
);

此scheduled_for是可更改的。这也是一个具体的例子

Hao article 2016-01-12 02:09:00+0800 3930462206848285
Hao article 2016-01-12 03:09:00+0801 3930462206848285
Hao article 2016-01-12 04:09:00+0802 3930462206848285
Hao article 2016-01-12 05:09:00+0803 3930462206848285

感谢您的回复！

Answer 1

为什么不使用insert if not exists？

https://docs.datastax.com/en/cql/3.1/cql/cql_reference/insert_r.html

为什么我的cassandra数据库中的数据插入速度如此之慢？

1 个答案: