Cassandra延迟/拒绝更新

时间:2014-05-01 21:07:39

标签: cassandra cql3

我遇到了一个曾经运作良好的小型Cassandra集群的问题。我曾经有3个节点。当我添加第4个时,我开始看到一些值没有更新的问题,所以我在整个集群上进行了nodetool修复(现在几次)。我应该提到我在从python-cql升级到新的python cassandra驱动程序的同时进行了切换。

基本上奇怪的是两种情况: 拒绝更新:

cqlsh:analytics> select * from metrics where id = '36122cc69a7a12e266ab40f5b7756daee75bd0d2735a707b369302acb879eedc';

(0 rows)


Tracing session: 19e5bbd0-d172-11e3-a039-67dcdc0d02de

 activity                                                                                                                 | timestamp    | source         | source_elapsed
--------------------------------------------------------------------------------------------------------------------------+--------------+----------------+----------------
                                                                                                   execute_cql3_query | 20:49:34,221 | 10.128.214.245 |              0
 Parsing select * from metrics where id = '36122cc69a7a12e266ab40f5b7756daee75bd0d2735a707b369302acb879eedc' LIMIT 10000; | 20:49:34,221 | 10.128.214.245 |            176
                                                                                                  Preparing statement | 20:49:34,222 | 10.128.214.245 |            311
                                                                                   Sending message to /10.128.180.108 | 20:49:34,222 | 10.128.214.245 |            773
                                                                                Message received from /10.128.214.245 | 20:49:34,224 | 10.128.180.108 |             67
                                                                                                        Row cache hit | 20:49:34,225 | 10.128.180.108 |            984
                                                                                   Read 0 live and 0 tombstoned cells | 20:49:34,225 | 10.128.180.108 |           1079
                                                                                Message received from /10.128.180.108 | 20:49:34,227 | 10.128.214.245 |           5760
                                                                                Enqueuing response to /10.128.214.245 | 20:49:34,227 | 10.128.180.108 |           3045
                                                                             Processing response from /10.128.180.108 | 20:49:34,227 | 10.128.214.245 |           5942
                                                                                   Sending message to /10.128.214.245 | 20:49:34,227 | 10.128.180.108 |           3302
                                                                                                     Request complete | 20:49:34,227 | 10.128.214.245 |           6282


cqlsh:analytics> update metrics set n = n + 1 where id = '36122cc69a7a12e266ab40f5b7756daee75bd0d2735a707b369302acb879eedc';

Tracing session: 20845ff0-d172-11e3-a039-67dcdc0d02de

 activity                                                                                                            | timestamp    | source         | source_elapsed
---------------------------------------------------------------------------------------------------------------------+--------------+----------------+----------------
                                                                                              execute_cql3_query | 20:49:45,328 | 10.128.214.245 |              0
 Parsing update metrics set n = n + 1 where id = '36122cc69a7a12e266ab40f5b7756daee75bd0d2735a707b369302acb879eedc'; | 20:49:45,328 | 10.128.214.245 |            129
                                                                                             Preparing statement | 20:49:45,328 | 10.128.214.245 |            227
                                                                               Determining replicas for mutation | 20:49:45,328 | 10.128.214.245 |            298
                                                                      Enqueuing counter update to /10.128.194.70 | 20:49:45,328 | 10.128.214.245 |            425
                                                                               Sending message to /10.128.194.70 | 20:49:45,329 | 10.128.214.245 |            598
                                                                           Message received from /10.128.214.245 | 20:49:45,330 |  10.128.194.70 |             37
                                                                                  Acquiring switchLock read lock | 20:49:45,331 |  10.128.194.70 |            623
                                                                            Message received from /10.128.194.70 | 20:49:45,331 | 10.128.214.245 |           3335
                                                                                          Appending to commitlog | 20:49:45,331 |  10.128.194.70 |            645
                                                                         Processing response from /10.128.194.70 | 20:49:45,331 | 10.128.214.245 |           3431
                                                                                      Adding to metrics memtable | 20:49:45,331 |  10.128.194.70 |            692
                                                                              Sending message to /10.128.214.245 | 20:49:45,332 |  10.128.194.70 |           1120
                                                                                                  Row cache miss | 20:49:45,332 |  10.128.194.70 |           1611
                                                                     Executing single-partition query on metrics | 20:49:45,332 |  10.128.194.70 |           1687
                                                                                    Acquiring sstable references | 20:49:45,332 |  10.128.194.70 |           1692
                                                                                     Merging memtable tombstones | 20:49:45,332 |  10.128.194.70 |           1692
                                                                                 Key cache hit for sstable 13958 | 20:49:45,332 |  10.128.194.70 |           1714
                                                                     Seeking to partition beginning in data file | 20:49:45,332 |  10.128.194.70 |           1856
                                                                                 Key cache hit for sstable 14036 | 20:49:45,333 |  10.128.194.70 |           2271
                                                                     Seeking to partition beginning in data file | 20:49:45,333 |  10.128.194.70 |           2271
                                       Skipped 0/2 non-slice-intersecting sstables, included 0 due to tombstones | 20:49:45,333 |  10.128.194.70 |           2540
                                                                      Merging data from memtables and 2 sstables | 20:49:45,333 |  10.128.194.70 |           2564
                                                                              Read 0 live and 1 tombstoned cells | 20:49:45,333 |  10.128.194.70 |           2632
                                                                              Sending message to /10.128.195.149 | 20:49:45,335 |  10.128.194.70 |           null
                                                                            Message received from /10.128.194.70 | 20:49:45,335 | 10.128.180.108 |             43
                                                                              Sending message to /10.128.180.108 | 20:49:45,335 |  10.128.194.70 |           null
                                                                                  Acquiring switchLock read lock | 20:49:45,335 | 10.128.180.108 |            297
                                                                                          Appending to commitlog | 20:49:45,335 | 10.128.180.108 |            312
                                                                            Message received from /10.128.194.70 | 20:49:45,336 | 10.128.195.149 |             53
                                                                                      Adding to metrics memtable | 20:49:45,336 | 10.128.180.108 |            374
                                                                            Enqueuing response to /10.128.194.70 | 20:49:45,336 | 10.128.180.108 |            445
                                                                               Sending message to /10.128.194.70 | 20:49:45,336 | 10.128.180.108 |            677
                                                                           Message received from /10.128.180.108 | 20:49:45,337 |  10.128.194.70 |           null
                                                                        Processing response from /10.128.180.108 | 20:49:45,337 |  10.128.194.70 |           null
                                                                                  Acquiring switchLock read lock | 20:49:45,338 | 10.128.195.149 |           1874
                                                                                          Appending to commitlog | 20:49:45,338 | 10.128.195.149 |           1970
                                                                                      Adding to metrics memtable | 20:49:45,338 | 10.128.195.149 |           2027
                                                                            Enqueuing response to /10.128.194.70 | 20:49:45,338 | 10.128.195.149 |           2147
                                                                               Sending message to /10.128.194.70 | 20:49:45,338 | 10.128.195.149 |           2572
                                                                           Message received from /10.128.195.149 | 20:49:45,339 |  10.128.194.70 |           null
                                                                        Processing response from /10.128.195.149 | 20:49:45,339 |  10.128.194.70 |           null
                                                                                                Request complete | 20:49:45,331 | 10.128.214.245 |           3556


cqlsh:analytics> select * from metrics where id = '36122cc69a7a12e266ab40f5b7756daee75bd0d2735a707b369302acb879eedc';

(0 rows)


Tracing session: 28f1f7b0-d172-11e3-a039-67dcdc0d02de

 activity                                                                                                                 | timestamp    | source         | source_elapsed
--------------------------------------------------------------------------------------------------------------------------+--------------+----------------+----------------
                                                                                                   execute_cql3_query | 20:49:59,468 | 10.128.214.245 |              0
 Parsing select * from metrics where id = '36122cc69a7a12e266ab40f5b7756daee75bd0d2735a707b369302acb879eedc' LIMIT 10000; | 20:49:59,468 | 10.128.214.245 |            119
                                                                                                  Preparing statement | 20:49:59,468 | 10.128.214.245 |            235
                                                                                   Sending message to /10.128.180.108 | 20:49:59,468 | 10.128.214.245 |            574
                                                                                Message received from /10.128.214.245 | 20:49:59,469 | 10.128.180.108 |             49
                                                                                                       Row cache miss | 20:49:59,470 | 10.128.180.108 |            817
                                                                          Executing single-partition query on metrics | 20:49:59,470 | 10.128.180.108 |            877
                                                                                         Acquiring sstable references | 20:49:59,470 | 10.128.180.108 |            888
                                                                                          Merging memtable tombstones | 20:49:59,470 | 10.128.180.108 |            938
                                                                                       Key cache hit for sstable 5399 | 20:49:59,470 | 10.128.180.108 |           1025
                                                                          Seeking to partition beginning in data file | 20:49:59,470 | 10.128.180.108 |           1033
                                                                                Message received from /10.128.180.108 | 20:49:59,471 | 10.128.214.245 |           3378
                                            Skipped 0/3 non-slice-intersecting sstables, included 0 due to tombstones | 20:49:59,471 | 10.128.180.108 |           1495
                                                                             Processing response from /10.128.180.108 | 20:49:59,471 | 10.128.214.245 |           3466
                                                                           Merging data from memtables and 1 sstables | 20:49:59,471 | 10.128.180.108 |           1507
                                                                                   Read 0 live and 1 tombstoned cells | 20:49:59,471 | 10.128.180.108 |           1660
                                                                                   Read 0 live and 0 tombstoned cells | 20:49:59,471 | 10.128.180.108 |           1759
                                                                                Enqueuing response to /10.128.214.245 | 20:49:59,471 | 10.128.180.108 |           1817
                                                                                   Sending message to /10.128.214.245 | 20:49:59,471 | 10.128.180.108 |           1977
                                                                                                     Request complete | 20:49:59,471 | 10.128.214.245 |           3638

所以这很简单。从阅读开始,它可能与墓碑时间戳有关,这些时间戳会以某种方式搞砸了。然而,它已经过了好几天了,现在"现在"仍然没有赶上墓碑的未来时间戳。有没有办法在活动停止时将整个群集的每个时间戳重置为0,我现在可以使用稍微不真实的数据。

第二个问题出在我的一些表格上。有些表会立即反映更改,但对于其他表,更改将在30分钟到1小时之间反映出来。我无法弄清楚时间戳与此有何关系。

我已经使用NTP同步了我的群集中的所有节点,而不是最精确,但在天或任何事情的规模上都不会失去同步。所有节点从一开始就已经同步,没有任何时候我没有完全不同步的时间。

有人可以帮忙吗?就像我说的那样,到目前为止,我已经决定关闭对集群的访问权限并将所有时间戳重置为0,我并不关心让一些订单错误,我只想要这样做< EM>工作

由于

1 个答案:

答案 0 :(得分:1)

时间戳是不可变的。您必须截断表并重建它。重建的最简单方法是插入正确的数据,但如果这不是一个选项,你可以通过sstable2json往返 - &gt;编辑时间戳 - &gt; json2sstable。