Question

我使用Batch将数据插入Cassandra。当我开始工作时，我正处于异常之下。

caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Batch too large   
  at com.datastax.driver.core.Responses$Error.asException(Responses.java:136)   
  at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:179)     
  at com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:184)    
  at com.datastax.driver.core.RequestHandler.access$2500(RequestHandler.java:43)    
  at com.datastax.driver.core.RequestHandler$SpeculativeExecution.setFinalResult(RequestHandler.java:798)   
  at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:617)    
  at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1005)  
  at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:928)   
  at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)     
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)   
  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)     
  at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)     
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)   
  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)

我在这个问题上已经阅读了很多博客。但这没有用。我在初始化时尝试将“spark.cassandra.output.batch.size.bytes”设置为spark conf。仍然没有解决我的问题。我得到了同样的错误。我的批处理有大约1000个插入语句。

请在下面找到我的代码。

CassandraConnector connector = CassandraConnector.apply(javaSparkContext.getConf());  
pairRDD.mapToPair(earnCalculatorKeyIterableTuple2 -> {  
            if (condition) {  
                do something......
            }  
            else {  
                Session session = connector.openSession();  
                BatchStatement batch = new   BatchStatement(BatchStatement.Type.UNLOGGED);            batch.setConsistencyLevel(ConsistencyLevel.valueOf(LOCAL_QUOROM));  
                PreparedStatement statement = session.prepare('my insert query');  
                for (condition) {  
                    if (!condition) {  
                        break;  
                    }  
                    Tuple2._2.forEach(s -> {  
                        if (!condition) {  
                            LOG.info(message);  
                        }  
                        else {  
                            BoundStatement boundStatement = statement.bind("bind variables");  
                            batch.add(boundStatement);  
                        }  
                    });  
                    session.execute(batch);  
                    batch.clear();  
                }  
                session.close();  
            }  
            return Tuple2;  
        });  
        return s;  
    }

感谢任何帮助。

Answer 1

您手动创建批次，批次太大。为每个批次添加较少的行。有很多方法可以手动执行此操作，但最简单的方法是添加一个计数器，每次添加X语句时都会提交批处理？

您要更改的参数仅与saveToCassandra完成的自动批处理有关。

Cassandra Batch InvalidQueryException - 批处理太大

1 个答案: