Cassandra另一个帧大小问题

时间：2015-01-05 16:55:38

标签： encryption cassandra pentaho thrift kettle

我知道Apache Cassandra的这种问题就在不久前，例如： Pentaho Frame size (17727647) larger than max length (16384000)!和 thrift_max_message_length_in_mb not recognized Cassandra

但我的问题有点不同。我使用Cassandra v2.0.7，我想从Pentaho Kettle（5.1）插入数据到Cassandra。我必须启用加密（SSL），所以我查了cassandra.yaml 

client_encryption_options:
enabled: true
keystore: /usr/cassandra/cassandra/conf/.keystore
keystore_password: password
require_client_auth: true
# Set trustore and truststore_password if require_client_auth is true
truststore: /usr/cassandra/cassandra/conf/.truststore
truststore_password: password

现在，有一种奇怪的情况，因为当我禁用加密（启用：false）并通过Pentaho插入数据时一切正常。但是当我启用加密时，我得到了这个：

   2015/01/05 17:42:38 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : A problem occurred during initializ
ation of the step
2015/01/05 17:42:38 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : org.apache.thrift.transport.TTransp
ortException: Frame size (352518400) larger than max length (16384000)!
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:137)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.cassandra.thrift.Cassandra$Client.recv_set_cql_version(Cassandra.java:1855)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.cassandra.thrift.Cassandra$Client.set_cql_version(Cassandra.java:1842)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.CassandraConnection.checkOpen(CassandraConnection.java:159)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.CassandraConnection.setKeyspace(CassandraConnection.java:174)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.LegacyKeyspace.setKeyspace(LegacyKeyspace.java:100)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.CassandraConnection.getKeyspace(CassandraConnection.java:277)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.initialize(CassandraOutput.java:218)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.processRow(CassandraOutput.java:353)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2015/01/05 17:42:38 - Cassandra Output.0 -      at java.lang.Thread.run(Thread.java:745)
java.lang.NullPointerException
        at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.processRow(CassandraOutput.java:356)
        at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
        at java.lang.Thread.run(Thread.java:745)
2015/01/05 17:42:38 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Unexpected error
2015/01/05 17:42:38 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : java.lang.NullPointerException
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.processRow(CassandraOutput.java:356)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2015/01/05 17:42:38 - Cassandra Output.0 -      at java.lang.Thread.run(Thread.java:745)
child index = 1, logging object : org.pentaho.di.core.logging.LoggingObject@47406edf parent=277afd35-cec6-4972-a572-a68a58ff9ae7
2015/01/05 17:42:38 - t_product_rb - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Errors detected!
2015/01/05 17:42:38 - t_product_rb - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Errors detected!

我使用CQL3插入数据。 现在，我发现在cassandra.yaml中有两个属性： -thrift_framed_transport_size_in_mb和 -thrift_max_message_length_in_mb＆lt; - 但在我的文件中我没有这个

我用这个属性检查一些配置。一旦我添加了这个thrift_max_message_length_in_mb，然后我删除了这个等等。但每次我都遇到这个错误。

我注意到加密为我的框架添加了一些东西，但我不知道是什么以及如何。 有人知道如何解决这个问题吗？

有关此问题的其他一些信息： https://issues.apache.org/jira/browse/THRIFT-1324 https://issues.apache.org/jira/browse/THRIFT-1323

EDIT 我再次注意到当我将端口从9160更改为9042（使用CQL3）时，我得到了不同的错误。

2015/01/09 12:53:21 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : A problem occurred during initializ
ation of the step
2015/01/09 12:53:21 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : org.apache.thrift.transport.TTransp
ortException: java.net.SocketTimeoutException: Read timed out
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.cassandra.thrift.Cassandra$Client.recv_set_cql_version(Cassandra.java:1855)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.cassandra.thrift.Cassandra$Client.set_cql_version(Cassandra.java:1842)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.CassandraConnection.checkOpen(CassandraConnection.java:159)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.CassandraConnection.setKeyspace(CassandraConnection.java:174)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.LegacyKeyspace.setKeyspace(LegacyKeyspace.java:100)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.CassandraConnection.getKeyspace(CassandraConnection.java:277)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.initialize(CassandraOutput.java:218)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.processRow(CassandraOutput.java:353)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.lang.Thread.run(Thread.java:745)
2015/01/09 12:53:21 - Cassandra Output.0 - Caused by: java.net.SocketTimeoutException: Read timed out
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.net.SocketInputStream.socketRead0(Native Method)
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.net.SocketInputStream.read(SocketInputStream.java:152)
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.net.SocketInputStream.read(SocketInputStream.java:122)
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
2015/01/09 12:53:21 - Cassandra Output.0 -      ... 18 more
java.lang.NullPointerException
        at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.processRow(CassandraOutput.java:356)
        at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
        at java.lang.Thread.run(Thread.java:745)
2015/01/09 12:53:21 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Unexpected error
2015/01/09 12:53:21 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : java.lang.NullPointerException
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.processRow(CassandraOutput.java:356)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.lang.Thread.run(Thread.java:745)
child index = 4, logging object : org.pentaho.di.core.logging.LoggingObject@6a4ea42b parent=c8d41836-602c-4a17-8977-62d691c419c5
2015/01/09 12:53:21 - Cassandra Output.0 - Finished processing (I=0, O=0, R=1, W=0, U=0, E=1)
2015/01/09 12:53:21 - t_customer_cm - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Errors detected!
2015/01/09 12:53:21 - t_customer_cm - Transformation detected one or more steps with errors.
2015/01/09 12:53:21 - t_customer_cm - Transformation is killing the other steps!
2015/01/09 12:53:21 - Dummy (do nothing).0 - Finished processing (I=0, O=0, R=10002, W=10002, U=0, E=0)
2015/01/09 12:53:21 - Get System Info.0 - Finished processing (I=0, O=0, R=20003, W=20003, U=0, E=0)
2015/01/09 12:53:21 - t_customer_cm - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Errors detected!

在cassandra上我得到了这个：

ERROR 12:53:11,807 Unexpected exception during request
 org.jboss.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 00000028800100010000000f7365745f63716c5f76657273696f6e000000010b0001000000
5332e302e3100
        at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:871)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

有什么想法吗？

2 个答案:

答案 0 :(得分：2)

Frame size (352518400) larger than max length (16384000)!
     

我注意到加密为我的框架添加了一些东西，但我不知道是什么以及如何。有人知道如何解决这个问题吗？

关于框架尺寸，它实际上并不重要，只有它有多大。正如布莱恩在THRIFT-1324中所写的那样，框架和非框架之间尺寸的唯一区别是＃34;是一个常量4个字节（int 32），在数据块之前并保持其大小。

理论解决方案对我来说非常明显：找出你的最大框架有多大并相应地配置设置。每个请求和每个响应都必须适合。

理论上，因为启用加密为您的数据增加了21倍的大小似乎很奇怪（您说它没有工作，因此它必须是＆lt; = 16384000）。这听起来很不寻常。

答案 1 :(得分：0)

检查以确保是否使用不安全连接CQL二进制协议（端口9042）。堆栈跟踪看起来表明正在使用Thrift，但CQL3使用完全不同的接口。是否有可能在使用SSL时使用thrift协议（端口9160），使用不安全时使用二进制协议（端口9042）？

编辑：这个答案可能不是一个好的答案。你提到使用CQL3让我觉得不应该涉及节俭，仍然可能值得一看，看看不安全的版本是使用本机二进制协议还是节俭。

相关问题
最新问题