Cassandra - 由于java.lang.OutOfMemoryError:直接缓冲区内存,节点无法重启

时间:2016-08-17 12:01:11

标签: cassandra datastax-java-driver

我正在使用最新的C* java driver提取20 GB的数据。 我有一个5节点C *集群。正在摄取数据的客户端应用程序正在不属于C *群集(但是属于同一LAN的一部分)的本地节点上运行。我也在我摄取的表上使用cassandra-lucene-index索引。 [cqlsh 5.0.1 | Cassandra 3.0.8 | CQL规范3.4.0 |原生协议v4]

我的Java客户端应用程序以下列方式工作:

  1. 解析json文件
  2. 将构建一个PreparedStatement
  3. 我文件中每个元素的一个BoundStatement被发送到带有executeAsync的C *集群[注意:有很多行正在发送]
  4. 大约一半时间,我的两个节点死亡,/var/log/cassandra/system.log显示

        ERROR [CompactionExecutor:1] JVMStabilityInspector.java:140 - JVM state determined to be unstable.  Exiting forcefully due to:
    java.lang.OutOfMemoryError: Direct buffer memory
            at java.nio.Bits.reserveMemory(Bits.java:693) ~[na:1.8.0_91]
            at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[na:1.8.0_91]
            at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[na:1.8.0_91]
            at org.apache.cassandra.utils.memory.BufferPool.allocate(BufferPool.java:108) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.utils.memory.BufferPool.access$1000(BufferPool.java:45) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.utils.memory.BufferPool$LocalPool.allocate(BufferPool.java:387) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.utils.memory.BufferPool$LocalPool.access$000(BufferPool.java:314) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.utils.memory.BufferPool.takeFromPool(BufferPool.java:120) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.utils.memory.BufferPool.get(BufferPool.java:92) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.io.util.RandomAccessReader.allocateBuffer(RandomAccessReader.java:87) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.io.compress.CompressedRandomAccessReader.access$100(CompressedRandomAccessReader.java:38) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.io.compress.CompressedRandomAccessReader$Builder.createBuffer(CompressedRandomAccessReader.java:275) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:74) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:59) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.io.compress.CompressedRandomAccessReader$Builder.build(CompressedRandomAccessReader.java:283) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.io.util.CompressedSegmentedFile.createReader(CompressedSegmentedFile.java:145) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.io.util.SegmentedFile.createReader(SegmentedFile.java:133) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.io.sstable.format.SSTableReader.getFileDataInput(SSTableReader.java:1711) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.<init>(AbstractSSTableIterator.java:93) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.db.columniterator.SSTableIterator.<init>(SSTableIterator.java:46) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.db.columniterator.SSTableIterator.<init>(SSTableIterator.java:36) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:62) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:580) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:492) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at com.stratio.cassandra.lucene.IndexService.read(IndexService.java:618) ~[cassandra-lucene-index-plugin-3.0.8.0.jar:na]
            at com.stratio.cassandra.lucene.IndexWriterWide.finish(IndexWriterWide.java:89) ~[cassandra-lucene-index-plugin-3.0.8.0.jar:na]
            at org.apache.cassandra.index.SecondaryIndexManager$IndexGCTransaction.commit(SecondaryIndexManager.java:958) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.db.compaction.CompactionIterator$1$1.onMergedRows(CompactionIterator.java:197) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:484) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:446) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:220) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:159) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:428) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:288) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:128) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:111) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:52) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:149) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:57) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263) ~[apache-cassandra-3.0.8.jar:3.0.8]
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_91]
            at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_91]
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_91]
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
            at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
    

    当我尝试重新启动两个节点时,会发生同样的异常。然后我决定调整MAX_HEAP_SIZE中的HEAP_NEWSIZEcassandra-env.sh,但节点只会耗尽所有可用内存,然后再次死亡。我有一个82gb的ram节点和一个48gb的ram死了。我为4G, 8G, 24G尝试了MAX_HEAP_SIZE,为800M, 2G尝试了HEAP_NEWSIZE,但他们无法重启。

    system.log文件还显示堆的转储:

    Heap dump file created
    
    INFO  [CompactionExecutor:1] 2016-08-14 13:43:44,223 HeapUtils.java:136 -
     num     #instances         #bytes  class name
    ----------------------------------------------
       1:         20879     2883199896  [I
       2:       5228765      334640960  org.apache.cassandra.utils.btree.BTreeSearchIterator
       3:       1836788      196567272  [B
       4:       3625993      163640704  [Ljava.lang.Object;
       5:       1260125       60486000  java.nio.HeapByteBuffer
       6:       1686322       53962304  org.apache.cassandra.utils.MergeIterator$Candidate
       7:        412664       52984560  [C
       8:        965525       38621000  org.apache.cassandra.db.rows.BufferCell
       9:       1511370       36272880  java.util.ArrayList
      10:        768459       30738360  org.apache.cassandra.db.rows.BTreeRow$Builder
      11:        382981       30638480  org.apache.cassandra.io.compress.CompressedRandomAccessReader
      12:        848641       27156512  java.util.RandomAccessSubList
      13:        848467       27150944  java.util.AbstractList$ListItr
      14:        394468       25245952  java.nio.DirectByteBuffer
      15:        387495       24799680  java.nio.DirectByteBufferR
      16:        774559       24785888  org.apache.cassandra.utils.btree.BTree$Builder
      17:        382823       24500672  org.apache.cassandra.db.columniterator.SSTableIterator$ForwardIndexedReader
      18:        486500       23352000  org.apache.cassandra.db.columniterator.SSTableIterator
      19:        383067       21451752  org.apache.cassandra.db.ClusteringPrefix$Deserializer
      20:        667956       21374592  org.apache.cassandra.db.rows.BTreeRow
      21:        853186       20476464  java.util.Arrays$ArrayList
      22:        848462       20363088  java.util.SubList$1
      23:        486689       19467560  org.apache.cassandra.db.rows.SerializationHelper
      24:        486319       19452760  org.apache.cassandra.db.filter.ClusteringIndexNamesFilter$1
      25:        383067       18387216  org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer
      26:        404602       16184080  org.apache.cassandra.utils.MergeIterator$ManyToOne
      27:        487237       15591584  java.util.TreeMap$KeyIterator
      28:        382823       15312920  org.apache.cassandra.db.columniterator.AbstractSSTableIterator$IndexState
      29:        376176       15047040  sun.misc.Cleaner
      30:        469635       15028320  org.apache.cassandra.db.rows.EncodingStats
      31:        464757       14872224  com.google.common.collect.Iterators$11
      32:        404602       13837016  [Lorg.apache.cassandra.utils.MergeIterator$Candidate;
      33:        550947       13222728  java.lang.Long
      34:        404502       12598432  [Lorg.apache.cassandra.db.rows.Row;
      35:        768640       12298240  org.apache.cassandra.db.rows.BTreeRow$Builder$CellResolver
      36:        375337       12010784  java.nio.DirectByteBuffer$Deallocator
      37:        250019       10000760  org.apache.cassandra.db.rows.Row$Merger
      38:        250019       10000760  org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer
      39:        312161        9989152  com.google.common.collect.RegularImmutableList
      40:        154609        9894976  org.apache.cassandra.db.SinglePartitionReadCommand
      41:        383092        9194208  org.apache.cassandra.cache.KeyCacheKey
      42:        275673        8821536  java.util.ArrayList$Itr
      43:        155947        8733032  java.util.LinkedHashMap
      44:        344055        8257320  java.lang.Double
      45:        157837        7576176  java.util.TreeMap
      46:        314112        7538688  com.google.common.collect.Iterators$12
      47:        302346        7256304  com.google.common.collect.Collections2$TransformedCollection
      48:        125865        7048440  java.util.stream.ReferencePipeline$Head
      49:        125057        7003192  org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator
      50:        154483        6179320  com.stratio.cassandra.lucene.IndexWriterWide
      51:        383788        6140608  java.util.zip.CRC32
      52:        377473        6039568  java.lang.Integer
      53:        250019        6000456  org.apache.cassandra.db.rows.Row$Merger$CellReducer
      54:        236029        5844680  [Ljava.nio.ByteBuffer;
      55:        231113        5546712  java.lang.String
      56:        230392        5529408  org.apache.cassandra.db.Clustering
      57:        125057        5002280  org.apache.cassandra.db.rows.RangeTombstoneMarker$Merger
      58:        154949        4958368  org.apache.cassandra.db.filter.ColumnFilter
      59:        309592        4953472  com.google.common.collect.Iterables$3
      60:        154483        4943456  [Lorg.apache.cassandra.db.rows.ColumnData;
      61:        154483        4943456  [Lorg.apache.cassandra.db.rows.Row$Builder;
      62:        154483        4943456  org.apache.cassandra.db.rows.Rows$1
      63:        179483        4307592  com.google.common.collect.Iterators$5
      64:          8055        4213168  [J
      65:        125114        4003648  java.util.ArrayList$ArrayListSpliterator
      66:        125057        4001824  org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer
      67:        125060        3828904  [Lorg.apache.cassandra.db.DeletionTime;
      68:        125057        3828808  [Lorg.apache.cassandra.db.rows.RangeTombstoneMarker;
      69:        157055        3769320  com.google.common.collect.Iterables$2
      70:        155783        3738792  org.apache.cassandra.utils.Interval
      71:        154722        3713328  org.apache.cassandra.db.rows.ComplexColumnData
      72:        154677        3712248  org.apache.cassandra.db.filter.ClusteringIndexNamesFilter
      73:        154483        3707592  org.apache.cassandra.index.SecondaryIndexManager$IndexGCTransaction$1
      74:        150356        3608544  org.apache.cassandra.db.Columns
      75:         70425        3380400  [Lorg.apache.cassandra.db.ClusteringPrefix$Kind;
      76:        124989        2999736  org.apache.cassandra.db.rows.ComplexColumnData$Builder
      77:        124971        2999304  java.util.stream.MatchOps$$Lambda$105/1652649396
      78:        124971        2999304  java.util.stream.MatchOps$1MatchSink
      79:        124971        2999304  java.util.stream.MatchOps$MatchOp
      80:        173871        2781936  org.apache.cassandra.db.rows.CellPath$CollectionCellPath
      81:        155081        2481296  java.util.TreeMap$KeySet
      82:        154648        2474368  java.util.TreeSet
      83:        154482        2471712  com.stratio.cassandra.lucene.IndexWriterWide$$Lambda$146/22703726
      84:         16388        2228768  com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$PaddedAtomicReference
      85:         54841        1754912  java.util.HashMap$Node
      86:         36123        1444920  sun.nio.cs.UTF_8$Decoder
      87:         35748        1429920  org.apache.cassandra.io.sstable.IndexHelper$IndexInfo
      88:         34554        1382160  java.util.TreeMap$Entry
      89:         11385        1126968  [Ljava.util.HashMap$Node;
      90:         18124        1014944  sun.nio.cs.UTF_8$Encoder
      91:         41388         993312  org.apache.cassandra.io.compress.CompressionMetadata$Chunk
      92:         37722         905328  java.lang.StringBuilder
      93:          7306         814736  java.lang.Class
      94:          8627         753872  [S
      95:         15574         747552  java.util.HashMap
      96:         14365         603008  [Ljava.lang.String;
      97:         23530         564720  java.util.EnumMap$EntryIterator$Entry
      98:         17382         556224  java.util.concurrent.ConcurrentHashMap$Node
      99:         22728         545472  org.apache.cassandra.db.ColumnFamilyStore$ViewFragment
     100:          6057         533016  java.lang.reflect.Method
     101:          8908         498848  org.apache.cassandra.utils.memory.BufferPool$Chunk
     102:         12317         450160  [Ljavax.management.ObjectName$Property;
     103:         18124         434976  javax.management.ObjectName$Property
     104:         10779         431160  java.util.LinkedHashMap$Entry
     105:         16300         391200  org.apache.cassandra.utils.btree.BTree$$Lambda$19/816798571
     106:          7827         375696  java.nio.HeapCharBuffer
     107:          9190         367600  java.lang.ref.Finalizer
     108:         16672         364336  [Ljava.lang.Class;
     109:          7538         361824  org.apache.cassandra.io.compress.CompressedRandomAccessReader$Builder
     110:         20623         329968  org.apache.cassandra.db.lifecycle.View$$Lambda$46/610085529
     111:          7491         299640  java.util.HashMap$KeyIterator
     112:          4551         291264  java.net.URL
     113:          9087         290784  sun.nio.fs.UnixPath
     114:           517         272976  [Lorg.apache.lucene.util.fst.FST$Arc;
     115:          7784         249088  java.io.File
     116:          3366         242352  java.lang.reflect.Field
     117:          3783         242112  java.util.regex.Matcher
     118:          4323         242088  java.util.zip.ZipFile$ZipFileInputStream
     119:          4322         242032  java.util.zip.ZipFile$ZipFileInflaterInputStream
     120:          9545         229080  java.util.concurrent.ConcurrentLinkedQueue$Node
     121:          9525         228600  com.google.common.collect.Iterators$8
     122:         14037         224592  java.lang.Object
     123:          3456         221184  io.netty.buffer.PoolSubpage
     124:          8942         214608  java.util.concurrent.CopyOnWriteArrayList$COWIterator
     125:          6702         214464  java.lang.StackTraceElement
     126:          2917         210024  org.apache.lucene.util.fst.FST$Arc
     127:          5119         204760  javax.management.MBeanPermission
     128:          6281         200992  java.io.FilePermission
     129:          4867         194680  javax.management.ObjectName
     130:          7807         187368  org.apache.cassandra.dht.Murmur3Partitioner$LongToken
     131:          7603         182472  org.apache.cassandra.io.util.MmappedRegions$Region
     132:          5624         179968  org.apache.cassandra.cql3.ColumnIdentifier
     133:          4330         173200  java.util.HashMap$ValueIterator
     134:          2901         162456  jdk.internal.org.objectweb.asm.Item
     135:          6604         158496  [Lorg.apache.cassandra.dht.Range;
     136:          6506         156152  [Ljava.security.ProtectionDomain;
     137:          1508         144768  java.util.jar.JarFile$JarFileEntry
     138:           323         140560  [Ljava.util.concurrent.ConcurrentHashMap$Node;
     139:          2143         137152  org.apache.lucene.store.ByteBufferIndexInput$SingleBufferImpl
     140:          5708         136992  org.apache.cassandra.dht.Token$KeyBound
     141:          4196         134272  org.apache.cassandra.db.rows.BTreeRow$$Lambda$140/1296076467
     142:          5445         130680  com.google.common.collect.Iterators$PeekingImpl
     143:          3990         127680  com.codahale.metrics.LongAdder
     144:          2273         127288  org.apache.cassandra.db.RowUpdateBuilder
     145:          1460         116800  java.util.zip.ZipEntry
     146:          4843         116232  java.util.concurrent.ConcurrentSkipListMap$Node
     147:          4837         116088  com.sun.jmx.mbeanserver.StandardMBeanSupport
     148:          3616         115712  java.util.Collections$UnmodifiableMap
     149:          4514         108336  org.apache.lucene.util.BytesRef
     150:          1931         108136  javax.management.MBeanServerNotification
     151:          2654         106160  java.security.AccessControlContext
     152:          4320         103680  com.sun.jmx.mbeanserver.NamedObject
     153:          4311         103464  org.apache.cassandra.utils.btree.BTree$FiltrationTracker
     154:          4295         103080  org.apache.cassandra.utils.Pair
     155:          6259         100144  java.io.FilePermission$1
     156:          2044          98112  org.apache.lucene.index.FieldInfo
     157:          2003          96144  org.antlr.runtime.CommonToken
     158:           106          95840  [Ljdk.internal.org.objectweb.asm.Item;
     159:          2902          92864  org.apache.cassandra.db.CBuilder$ArrayBackedBuilder
     160:          1022          89936  org.apache.lucene.codecs.blocktree.FieldReader
     161:          2779          88928  java.util.EnumMap$EntryIterator
     162:          5557          88912  [Lorg.apache.cassandra.db.Clustering;
     163:          2197          87880  org.apache.cassandra.utils.concurrent.Ref$State
     164:           908          87168  org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer$NumericEntry
     165:          1785          85680  com.codahale.metrics.EWMA
     166:          2063          82520  java.util.WeakHashMap$Entry
     167:          2470          81824  [Ljava.lang.reflect.Method;
     168:          2017          80680  java.util.HashMap$EntryIterator
     169:          3226          77424  java.io.ExpiringCache$Entry
     170:           875          77416  [Ljava.lang.StackTraceElement;
     171:           455          77408  [Z
     172:          2412          77184  java.io.FileDescriptor
     173:          3216          77184  java.util.concurrent.ConcurrentLinkedDeque$Node
     174:          2300          73600  com.google.common.collect.Iterators$7
     175:          1022          73584  org.apache.lucene.util.fst.FST
     176:          2283          73056  org.apache.lucene.util.WeakIdentityMap$IdentityWeakReference
     177:          3036          72864  org.apache.cassandra.db.LivenessInfo
     178:          2857          71792  [Lorg.yaml.snakeyaml.tokens.Token$ID;
     179:          1774          70960  java.util.Formatter$FormatSpecifier
     180:          1102          70528  sun.nio.ch.FileChannelImpl
     181:          4393          70288  java.util.HashMap$KeySet
     182:          2189          70048  org.apache.cassandra.metrics.CassandraMetricsRegistry$MetricName
     183:          2886          69264  java.net.URLClassLoader$1
     184:          2871          68904  org.apache.cassandra.db.DeletionTime
     185:          2853          68472  org.apache.cassandra.dht.Range
     186:           164          68224  [Lorg.antlr.runtime.BitSet;
     187:          1694          67760  java.lang.ref.SoftReference
     188:           128          67584  [Lcom.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$PaddedAtomicReference;
     189:          4196          67136  org.apache.cassandra.db.Columns$$Lambda$139/703415756
     190:          4175          66800  com.google.common.collect.Iterators$3
     191:          3938          63008  java.util.HashSet
     192:          2599          62376  com.google.common.collect.ImmutableEntry
     193:          1526          61040  org.apache.cassandra.db.PreHashedDecoratedKey
     194:          1891          60512  java.net.InetAddress$InetAddressHolder
     195:          3780          60480  com.stratio.cassandra.lucene.IndexWriterWide$$Lambda$149/2109187587
     196:           924          59136  java.util.concurrent.ConcurrentHashMap
     197:          1457          58280  java.lang.ClassNotFoundException
     198:          2411          57864  java.lang.StringBuffer
     199:          1797          57504  java.util.TreeMap$EntryIterator
     200:          1412          56480  java.util.EnumMap
     201:          2349          56376  java.util.Date
     202:          1563          56072  [Ljava.util.Formatter$Flags;
     203:          1390          55600  java.util.ArrayList$SubList$1
     204:           868          55552  java.text.DecimalFormatSymbols
     205:          1384          55360  org.yaml.snakeyaml.error.Mark
     206:           686          54880  java.lang.reflect.Constructor
     207:          1361          54440  java.util.ArrayList$SubList
     208:          2267          54408  org.apache.cassandra.metrics.CassandraMetricsRegistry$JmxGauge
     209:          2238          53712  sun.misc.ProxyGenerator$ConstantPool$IndirectEntry
     210:          2197          52728  org.apache.cassandra.utils.concurrent.Ref
     211:          2172          52128  java.util.Collections$1
     212:          1615          51680  java.util.concurrent.locks.AbstractQueuedSynchronizer$Node
     213:          2103          50472  org.apache.cassandra.db.lifecycle.View$$Lambda$66/2068475002
     214:          3151          50416  java.util.HashMap$Values
     215:           134          50384  java.lang.Thread
     216:           272          50048  [Lorg.apache.cassandra.net.MessagingService$Verb;
     217:          2042          49008  javax.management.ObjectInstance
     218:          1225          49000  java.lang.invoke.MethodType
     219:            61          48776  [D
     220:          2005          48120  java.util.Formatter$FixedString
     221:          1490          47680  java.security.CodeSource
     222:           569          47184  [Ljava.util.WeakHashMap$Entry;
     223:           208          46592  jdk.internal.org.objectweb.asm.MethodWriter
     224:           827          46312  java.lang.invoke.MemberName
     225:          1447          46304  sun.misc.URLClassPath$JarLoader$2
     226:          1916          45984  java.util.concurrent.atomic.AtomicLong
     227:          1896          45504  java.lang.Class$MethodArray
     228:          1886          45264  java.net.Inet4Address
     229:          1845          44280  javax.management.MBeanTrustPermission
     230:          1830          43920  [Lorg.apache.lucene.util.packed.PackedInts$Format;
     231:          1339          42848  java.lang.invoke.MethodType$ConcurrentWeakInternSet$WeakEntry
     232:           655          41920  java.util.stream.ReferencePipeline$3
     233:          1734          41616  com.google.common.collect.Iterables$6
     234:          2578          41248  com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$1
     235:          2560          40960  com.google.common.collect.Maps$6
     236:          1022          40880  org.apache.lucene.util.fst.BytesStore
     237:           842          40416  org.apache.cassandra.metrics.CassandraMetricsRegistry$JmxTimer
     238:          1253          40096  java.lang.ref.ReferenceQueue
     239:           703          39368  java.util.concurrent.ConcurrentHashMap$EntryIterator
     240:           264          38016  com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$PaddedAtomicLong
     241:          1539          36936  org.apache.lucene.util.fst.ReverseBytesReader
     242:           655          36680  java.lang.Class$ReflectionData
     243:           648          36288  java.util.concurrent.ConcurrentHashMap$KeyIterator
     244:          2111          33776  java.util.concurrent.atomic.AtomicInteger
     245:           421          33680  org.apache.cassandra.db.rows.RowAndDeletionMergeIterator
     246:          2092          33472  org.apache.cassandra.utils.concurrent.Refs
     247:          1022          32704  org.apache.lucene.store.ByteArrayDataInput
     248:          1358          32592  org.apache.cassandra.gms.GossipDigest
     249:           789          31560  sun.nio.ch.FileChannelImpl$Unmapper
     250:           656          31488  java.util.StringTokenizer
    ...
    3851:             1             16  sun.util.locale.provider.SPILocaleProviderAdapter
    3852:             1             16  sun.util.resources.LocaleData
    3853:             1             16  sun.util.resources.LocaleData$LocaleDataResourceBundleControl
    Total      41014224     4650350712
    
    1. 如何让它们重新开始? 我觉得他们试图重新构建一些排队的插入,因为我只是从我的客户端“垃圾”插入了writeAsync,而cassandra最终被插入的数量所淹没?

    2. 下次我提取数据时如何防止这种情况发生?

1 个答案:

答案 0 :(得分:1)

这是由于Java的较早版本(1.8.102和1.9.104之前的版本)对此没有限制而导致对堆缓冲区使用过多的内存引起的:https://support.datastax.com/hc/en-us/articles/360000863663-JVM-OOM-direct-buffer-errors-affected-by-unlimited-java-nio-cache

尝试升级到较新的Java并添加:

-Djdk.nio.maxCachedBufferSize=1048576

cassandra-env.shjvm.options