cassandra sstableloader挂着接收进度100%

时间:2015-12-19 02:24:37

标签: cassandra datastax cql cassandra-2.0

我试图在cassandra 2.0.7中通过sstableloader加载数据。 终端显示进度100%。我按nodetool netstats检查netstats 它显示:

Mode: NORMAL
Bulk Load 21d7d610-a5f2-11e5-baa7-8fc95be03ac4
    /10.19.150.70
        Receiving 4 files, 138895248 bytes total
            /root/data/whatyyun/metadata/whatyyun-metadata-tmp-jb-8-Data.db 67039680/67039680 bytes(100%) received from /10.19.150.70
            /root/data/whatyyun/metadata/whatyyun-metadata-tmp-jb-10-Data.db 3074549/3074549 bytes(100%) received from /10.19.150.70
            /root/data/whatyyun/metadata/whatyyun-metadata-tmp-jb-9-Data.db 43581052/43581052 bytes(100%) received from /10.19.150.70
            /root/data/whatyyun/metadata/whatyyun-metadata-tmp-jb-7-Data.db 25199967/25199967 bytes(100%) received from /10.19.150.70
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool Name                    Active   Pending      Completed
Commands                        n/a         0              0
Responses                       n/a         0          11671

sstableloader挂了好几个小时。我检查日志是否存在可能存在的错误。

ERROR [CompactionExecutor:7] 2015-12-19 09:45:53,811 CassandraDaemon.java (line 198) Exception in thread Thread[CompactionExecutor:7,1,main]
java.lang.IndexOutOfBoundsException
        at java.nio.Buffer.checkIndex(Buffer.java:532)
        at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
        at org.apache.cassandra.db.marshal.TimeUUIDType.compareTimestampBytes(TimeUUIDType.java:62)
        at org.apache.cassandra.db.marshal.TimeUUIDType.compare(TimeUUIDType.java:51)
        at org.apache.cassandra.db.marshal.TimeUUIDType.compare(TimeUUIDType.java:31)
        at org.apache.cassandra.dht.LocalToken.compareTo(LocalToken.java:44)
        at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:85)
        at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:36)
        at java.util.concurrent.ConcurrentSkipListMap.findNode(ConcurrentSkipListMap.java:804)
        at java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:828)
        at java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1626)
        at org.apache.cassandra.db.Memtable.resolve(Memtable.java:215)
        at org.apache.cassandra.db.Memtable.put(Memtable.java:173)
        at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:893)
        at org.apache.cassandra.db.index.AbstractSimplePerColumnSecondaryIndex.insert(AbstractSimplePerColumnSecondaryIndex.java:107)
        at org.apache.cassandra.db.index.SecondaryIndexManager.indexRow(SecondaryIndexManager.java:441)
        at org.apache.cassandra.db.Keyspace.indexRow(Keyspace.java:407)
        at org.apache.cassandra.db.index.SecondaryIndexBuilder.build(SecondaryIndexBuilder.java:62)
        at org.apache.cassandra.db.compaction.CompactionManager$9.run(CompactionManager.java:833)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
ERROR [NonPeriodicTasks:1] 2015-12-19 09:45:53,812 CassandraDaemon.java (line 198) Exception in thread Thread[NonPeriodicTasks:1,5,main]
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException
        at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413)
        at org.apache.cassandra.db.index.SecondaryIndexManager.maybeBuildSecondaryIndexes(SecondaryIndexManager.java:142)
        at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:113)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:188)
        at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409)
        ... 9 more
Caused by: java.lang.IndexOutOfBoundsException
        at java.nio.Buffer.checkIndex(Buffer.java:532)
        at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
        at org.apache.cassandra.db.marshal.TimeUUIDType.compareTimestampBytes(TimeUUIDType.java:62)
        at org.apache.cassandra.db.marshal.TimeUUIDType.compare(TimeUUIDType.java:51)
        at org.apache.cassandra.db.marshal.TimeUUIDType.compare(TimeUUIDType.java:31)
        at org.apache.cassandra.dht.LocalToken.compareTo(LocalToken.java:44)
        at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:85)
        at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:36)
        at java.util.concurrent.ConcurrentSkipListMap.findNode(ConcurrentSkipListMap.java:804)
        at java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:828)
        at java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1626)
        at org.apache.cassandra.db.Memtable.resolve(Memtable.java:215)
at java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1626)
        at org.apache.cassandra.db.Memtable.resolve(Memtable.java:215)
        at org.apache.cassandra.db.Memtable.put(Memtable.java:173)
        at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:893)
        at org.apache.cassandra.db.index.AbstractSimplePerColumnSecondaryIndex.insert(AbstractSimplePerColumnSecondaryIndex.java:107)
        at org.apache.cassandra.db.index.SecondaryIndexManager.indexRow(SecondaryIndexManager.java:441)
        at org.apache.cassandra.db.Keyspace.indexRow(Keyspace.java:407)
        at org.apache.cassandra.db.index.SecondaryIndexBuilder.build(SecondaryIndexBuilder.java:62)
        at org.apache.cassandra.db.compaction.CompactionManager$9.run(CompactionManager.java:833)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        ... 3 more

该表的方案如下:

CREATE TABLE metadata (
  userid timeuuid,
  dirname text,
  basename text,
  ctime timestamp,
  fileid timeuuid,
  imagefileid timeuuid,
  imagefilesize int,
  mtime timestamp,
  nodetype int,
  showname text,
  size bigint,
  timelong text,
  PRIMARY KEY (userid, dirname, basename, ctime)
) WITH
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='' AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=864000 AND
  index_interval=128 AND
  read_repair_chance=0.100000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  default_time_to_live=0 AND
  speculative_retry='99.0PERCENTILE' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'LZ4Compressor'};

CREATE INDEX idx_fileid ON metadata (fileid);

CREATE INDEX idx_nodetype ON metadata (nodetype);

我可以安全地杀死sstableloader的进程吗?这个批量加载过程完成了吗?

3 个答案:

答案 0 :(得分:1)

您应该尝试增加SSTABLELoader的堆。

vim $(哪个sstableloader)

#########"$JAVA" $JAVA_AGENT -ea -cp "$CLASSPATH" $JVM_OPTS -Xmx$MAX_HEAP_SIZE \
"$JAVA" $JAVA_AGENT -ea -cp "$CLASSPATH" $JVM_OPTS -XX:+UseG1GC -Xmx10G -Xms10G -XX:+UseTLAB -XX:+ResizeTLAB \
        -Dcassandra.storagedir="$cassandra_storagedir" \
        -Dlogback.configurationFile=logback-tools.xml \
        org.apache.cassandra.tools.BulkLoader "$@"

我希望能解决您的问题。

答案 1 :(得分:0)

您的节点必须耗尽资源可能是由于负载过重或任何其他进程。 尝试在高负载运行的节点上重新启动Cassandra,看看是否有帮助。

答案 2 :(得分:0)

从上述错误看来,您正在运行一些资源紧缩。因此,您需要调整一些内存方面的设置,例如Xmx,Xms,这些设置指示较低版本的Cassandra(即2.x)的cassandra-env.sh文件中的最小和最大堆大小

在进行上述调整之后,您需要重新启动节点/集群,然后尝试再次加载。