Cassandra OOM在内存中有大型RangeTombstoneList对象

时间:2018-01-25 12:28:41

标签: cassandra out-of-memory

我们的cassandra节点出现问题已经有一段时间了,所以最后我们对它们进行了配置,以便我们可以获得堆转储以尝试查看导致OOM的原因。

在转储中,有16个线程(名为SharedPool-Worker-XX),每个线程从同一个表执行SliceFromReadCommand。 16个线程中的每一个都有一个RangeTombstoneList对象,保留200到240mb的内存。

有问题的表被用作两个应用程序之间的队列(我知道,远非理想的cassandra用例,但就是这样),其中一个写入,另一个读取。因此,表格中不存在大量的墓碑......我怎么也找不到它们。

我对针对该表发出的查询进行了跟踪,结果如下:

cqlsh:ddp> select json_data
   ... from file_download
   ... where ti = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX';

 ti | uuid | json_data
----+------+-----------

(0 rows)

Tracing session: b2f2be60-01c8-11e8-ae90-fd18acdea80d

 activity                                                                                                                           | timestamp                  | source       | source_elapsed
------------------------------------------------------------------------------------------------------------------------------------+----------------------------+--------------+----------------
                                                                                                             Execute CQL3 query | 2018-01-25 12:10:14.214000 | 10.60.73.232 |              0
 Parsing select json_data\nfrom file_download\nwhere ti = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'; [SharedPool-Worker-1] | 2018-01-25 12:10:14.260000 | 10.60.73.232 |            105
                                                                                      Preparing statement [SharedPool-Worker-1] | 2018-01-25 12:10:14.262000 | 10.60.73.232 |            197
                                                        Executing single-partition query on file_download [SharedPool-Worker-4] | 2018-01-25 12:10:14.263000 | 10.60.73.232 |            442
                                                                             Acquiring sstable references [SharedPool-Worker-4] | 2018-01-25 12:10:14.264000 | 10.60.73.232 |            491
                                                                              Merging memtable tombstones [SharedPool-Worker-4] | 2018-01-25 12:10:14.265000 | 10.60.73.232 |            517
                                                                Bloom filter allows skipping sstable 2444 [SharedPool-Worker-4] | 2018-01-25 12:10:14.270000 | 10.60.73.232 |            608
                                                                   Bloom filter allows skipping sstable 8 [SharedPool-Worker-4] | 2018-01-25 12:10:14.271000 | 10.60.73.232 |            665
                                Skipped 0/2 non-slice-intersecting sstables, included 0 due to tombstones [SharedPool-Worker-4] | 2018-01-25 12:10:14.273000 | 10.60.73.232 |            700
                                                               Merging data from memtables and 0 sstables [SharedPool-Worker-4] | 2018-01-25 12:10:14.274000 | 10.60.73.232 |            707
                                                                        Read 0 live and 0 tombstone cells [SharedPool-Worker-4] | 2018-01-25 12:10:14.274000 | 10.60.73.232 |            754
                                                                                                               Request complete | 2018-01-25 12:10:14.215148 | 10.60.73.232 |           1148

cfstats还显示了avg./max tombstones pr。切片为0。

这让我怀疑OOM是否真的是由于大量的墓碑或其他原因造成的?

我们运行cassandra v2.1.17

0 个答案:

没有答案