cassandra,时间序列表的表现不佳

时间:2014-04-16 20:22:49

标签: cassandra

我有一个3x节点集群(在相同的16核心盒子上,通过lxc在虚拟盒子中,但是它上面的3TB磁盘上的每个节点都有)。

我的表是这样的:

CREATE TABLE history (
 id text,
 idx bigint,
 data bigint,
 PRIMARY KEY (id, idx)
) WITH CLUSTERING ORDER BY (idx DESC)

id将存储一个id为字符串的id,idx是以ms为单位的时间,数据是我的数据。根据我发现的所有示例,这似乎是时间序列数据的正确模式。

我的查询是:

选择idx,来自历史的数据,其中id =?限制2

这将返回最近的2个(基于idx)行。

由于id是分区键而idx是聚类键,因此我发现docs声称这对cassandra非常有效。但我的基准却说不然。

我已经填充了总共400GB(分成3个节点),现在我正在运行来自第二个盒子的查询。使用16或32个线程,我正在运行上述查询,但在3个不同磁盘上运行的3个节点的性能非常低:

throughput: 61         avg time: 614,808 μs
throughput: 57         avg time: 519,651 μs
throughput: 52         avg time: 569,245 μs

所以,每秒约55次查询,每次查询需要半秒钟(有时它们需要200毫秒)

我觉得这个很低。

有人可以告诉我我的架构是否正确以及是否建议架构?如果我的架构是正确的,我怎么能找到出错的地方?

16核盒子上的磁盘IO:

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda               0.00         0.00         0.00          0          0
sdb             135.00         6.76         0.00          6          0
sdc             149.00         6.99         0.00          6          0
sdd             124.00         7.21         0.00          7          0

cassandras不会使用超过1个cpu核心。

编辑:通过跟踪,当我为1个id运行一个简单的查询时,我会收到如下所示的很多行:

                                            Key cache hit for sstable 33259 | 20:16:26,699 | 127.0.0.1 |           5830
                                 Seeking to partition beginning in data file | 20:16:26,699 | 127.0.0.1 |           5833
                                  Bloom filter allows skipping sstable 33256 | 20:16:26,699 | 127.0.0.1 |           5923
                                  Bloom filter allows skipping sstable 33255 | 20:16:26,699 | 127.0.0.1 |           5932
                                  Bloom filter allows skipping sstable 33252 | 20:16:26,699 | 127.0.0.1 |           5938
                                             Key cache hit for sstable 33247 | 20:16:26,699 | 127.0.0.1 |           5948
                                 Seeking to partition beginning in data file | 20:16:26,699 | 127.0.0.1 |           5951
                                  Bloom filter allows skipping sstable 33246 | 20:16:26,699 | 127.0.0.1 |           6072
                                  Bloom filter allows skipping sstable 33243 | 20:16:26,699 | 127.0.0.1 |           6081
                                             Key cache hit for sstable 33242 | 20:16:26,699 | 127.0.0.1 |           6092
                                 Seeking to partition beginning in data file | 20:16:26,699 | 127.0.0.1 |           6095
                                  Bloom filter allows skipping sstable 33240 | 20:16:26,699 | 127.0.0.1 |           6187
                                             Key cache hit for sstable 33237 | 20:16:26,699 | 127.0.0.1 |           6198
                                 Seeking to partition beginning in data file | 20:16:26,699 | 127.0.0.1 |           6201
                                             Key cache hit for sstable 33235 | 20:16:26,699 | 127.0.0.1 |           6297
                                 Seeking to partition beginning in data file | 20:16:26,699 | 127.0.0.1 |           6301
                                  Bloom filter allows skipping sstable 33234 | 20:16:26,699 | 127.0.0.1 |           6393
                                             Key cache hit for sstable 33229 | 20:16:26,699 | 127.0.0.1 |           6404
                                 Seeking to partition beginning in data file | 20:16:26,699 | 127.0.0.1 |           6408
                                  Bloom filter allows skipping sstable 33228 | 20:16:26,699 | 127.0.0.1 |           6496
                                             Key cache hit for sstable 33227 | 20:16:26,699 | 127.0.0.1 |           6508
                                 Seeking to partition beginning in data file | 20:16:26,699 | 127.0.0.1 |           6511
                                             Key cache hit for sstable 33226 | 20:16:26,699 | 127.0.0.1 |           6601
                                 Seeking to partition beginning in data file | 20:16:26,699 | 127.0.0.1 |           6605
                                             Key cache hit for sstable 33225 | 20:16:26,700 | 127.0.0.1 |           6692
                                 Seeking to partition beginning in data file | 20:16:26,700 | 127.0.0.1 |           6696
                                             Key cache hit for sstable 33223 | 20:16:26,700 | 127.0.0.1 |           6785
                                 Seeking to partition beginning in data file | 20:16:26,700 | 127.0.0.1 |           6789
                                             Key cache hit for sstable 33221 | 20:16:26,700 | 127.0.0.1 |           6876
                                 Seeking to partition beginning in data file | 20:16:26,700 | 127.0.0.1 |           6880
                                  Bloom filter allows skipping sstable 33219 | 20:16:26,700 | 127.0.0.1 |           6967
                                             Key cache hit for sstable 33377 | 20:16:26,700 | 127.0.0.1 |           6978
                                 Seeking to partition beginning in data file | 20:16:26,700 | 127.0.0.1 |           6981
                                             Key cache hit for sstable 33208 | 20:16:26,700 | 127.0.0.1 |           7071
                                 Seeking to partition beginning in data file | 20:16:26,700 | 127.0.0.1 |           7075
                                             Key cache hit for sstable 33205 | 20:16:26,700 | 127.0.0.1 |           7161
                                 Seeking to partition beginning in data file | 20:16:26,700 | 127.0.0.1 |           7166
                                  Bloom filter allows skipping sstable 33201 | 20:16:26,700 | 127.0.0.1 |           7251
                                  Bloom filter allows skipping sstable 33200 | 20:16:26,700 | 127.0.0.1 |           7260
                                             Key cache hit for sstable 33195 | 20:16:26,700 | 127.0.0.1 |           7276
                                 Seeking to partition beginning in data file | 20:16:26,700 | 127.0.0.1 |           7279
                                  Bloom filter allows skipping sstable 33191 | 20:16:26,700 | 127.0.0.1 |           7363
                                             Key cache hit for sstable 33190 | 20:16:26,700 | 127.0.0.1 |           7374
                                 Seeking to partition beginning in data file | 20:16:26,700 | 127.0.0.1 |           7377
                                  Bloom filter allows skipping sstable 33189 | 20:16:26,700 | 127.0.0.1 |           7463
                                             Key cache hit for sstable 33186 | 20:16:26,700 | 127.0.0.1 |           7474
                                 Seeking to partition beginning in data file | 20:16:26,700 | 127.0.0.1 |           7477
                                             Key cache hit for sstable 33183 | 20:16:26,700 | 127.0.0.1 |           7563
                                 Seeking to partition beginning in data file | 20:16:26,700 | 127.0.0.1 |           7567
                                  Bloom filter allows skipping sstable 33182 | 20:16:26,701 | 127.0.0.1 |           7663
                                  Bloom filter allows skipping sstable 33180 | 20:16:26,701 | 127.0.0.1 |           7672
                                  Bloom filter allows skipping sstable 33178 | 20:16:26,701 | 127.0.0.1 |           7679
                                  Bloom filter allows skipping sstable 33177 | 20:16:26,701 | 127.0.0.1 |           7686

也许最重要的是跟踪结束:

                                Merging data from memtables and 277 sstables | 20:21:29,186 | 127.0.0.1 |         607001
                                          Read 3 live and 0 tombstoned cells | 20:21:29,186 | 127.0.0.1 |         607205
                                                            Request complete | 20:21:29,186 | 127.0.0.1 |         607714

1 个答案:

答案 0 :(得分:2)

请查看跟踪以确认,但如果sdb,sdc和sdd正在旋转磁盘,您将看到正确的tps数量级,并且很可能是读取端绑定的随机磁盘I / O. / p>

如果是这种情况,那么您只有两个选项(任何系统,不是特定于Cassandra):

  1. 切换到SSD。我的个人测试表明,当工作负载完全受磁盘的tps限制时,随机读取性能提高了3个数量级。
  2. 确保缓存了很大比例的读取。如果您正在对400GB数据进行随机读取,那可能不太可行。\
  3. Cassandra可以为每个CPU内核执行大约3k-5K的操作(读取或写入),但前提是磁盘子系统不是限制因素。