数据加载5m行后Cassandra查询失败 - 在一致性LOCAL_ONE读取查询期间Cassandra失败

时间：2016-10-06 14:15:10

标签： cassandra cql datastax-java-driver

问题

当我有大量数据加载时，简单CQL选择失败。

设置

我正在使用以下Cassandra架构：

CREATE KEYSPACE fv WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };


create table entity_by_identifier (
    identifier text,
    state entity_state,
    PRIMARY KEY(identifier)
);

CREATE TYPE entity_state,(
    identifier text,
    number1 int,
    number2 double,
    entity_type text,
    string1 text,
    string2 text
);

我正在尝试执行的查询：

SELECT * FROM fv.entity_by_identifier WHERE identifier=:identifier;

问题

此查询在小型数据集中正常工作（尝试使用500行）。 但是，通过大量数据负载测试，我在此表中创建了超过500万行，然后多次执行此查询（10个线程连续执行此查询1小时）。

数据加载完成后，查询开始但会立即失败，并显示以下错误：

com.datastax.driver.core.exceptions.ReadFailureException: Cassandra failure during read query at consistency LOCAL_ONE (1 responses were required but only 0 replica responded, 1 failed)
    at com.datastax.driver.core.exceptions.ReadFailureException.copy(ReadFailureException.java:85)
    at com.datastax.driver.core.exceptions.ReadFailureException.copy(ReadFailureException.java:27)
    at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
    at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
    at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:64)
...my calling classes...

我检查了Cassandra日志，发现只有这个例外：

java.lang.AssertionError: null
    at org.apache.cassandra.db.rows.BTreeRow.getCell(BTreeRow.java:212) ~[apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.db.SinglePartitionReadCommand.canRemoveRow(SinglePartitionReadCommand.java:899) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.db.SinglePartitionReadCommand.reduceFilter(SinglePartitionReadCommand.java:863) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndSSTablesInTimestampOrder(SinglePartitionReadCommand.java:748) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:519) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:496) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:358) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:366) ~[apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1797) ~[apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2466) ~[apache-cassandra-3.7.jar:3.7]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_101]
    at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.concurrent.SEPExecutor.maybeExecuteImmediately(SEPExecutor.java:192) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.service.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:117) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.service.AbstractReadExecutor.makeDataRequests(AbstractReadExecutor.java:85) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.service.AbstractReadExecutor$NeverSpeculatingReadExecutor.executeAsync(AbstractReadExecutor.java:214) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.doInitialQueries(StorageProxy.java:1702) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1657) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1604) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1523) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.db.SinglePartitionReadCommand.execute(SinglePartitionReadCommand.java:335) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.service.pager.AbstractQueryPager.fetchPage(AbstractQueryPager.java:67) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.service.pager.SinglePartitionPager.fetchPage(SinglePartitionPager.java:34) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.cql3.statements.SelectStatement$Pager$NormalPager.fetchPage(SelectStatement.java:325) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:361) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:237) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:78) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:208) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:486) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:463) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:130) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) [apache-cassandra-3.7.jar:3.7]
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.36.Final.jar:4.0.36.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292) [netty-all-4.0.36.Final.jar:4.0.36.Final]
    at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:32) [netty-all-4.0.36.Final.jar:4.0.36.Final]
    at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:283) [netty-all-4.0.36.Final.jar:4.0.36.Final]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_101]
    at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.7.jar:3.7]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]

正如您所看到的，我正在使用Cassandra 3.7。 正在使用的Datastax驱动程序是3.1.0版。

为什么较大的数据集可能会导致此错误？

2 个答案:

答案 0 :(得分：0)

对于要检索的记录数量，使用pagination来检索较小的块是值得的。

修改

正如here所解释的那样，您可能会遇到读取超时;浏览您引用的数百万条记录的时间可能超过read_request_timeout_in_ms阈值（默认为5 seconds）。一种选择是增加该阈值。

答案 1 :(得分：0)

找到解决问题的方法。

使用用户定义类型时，需要使用“冻结”关键字。

create table entity_by_identifier (
identifier text,
state entity_state,
PRIMARY KEY(identifier)
);

变为：

create table entity_by_identifier (
identifier text,
state frozen <entity_state>,
PRIMARY KEY(identifier)
);

您可以在以下位置找到有关此Frozen关键字的信息： http://docs.datastax.com/en/cql/3.1/cql/cql_using/cqlUseUDT.html 和 http://docs.datastax.com/en/cql/3.1/cql/cql_reference/create_table_r.html#reference_ds_v3f_vfk_xj__tuple-udt-columns

虽然，我仍然不清楚为什么缺少这个“冻结”关键字会导致我看到的错误。

相关问题
最新问题