Question

使用Cassandra 1.1.6，Pig 0.10.0和Hadoop 1.1.0，我可以在examples / pig中运行带有cassandra的pig_cassandra示例脚本。

但是当我改变时

rows = LOAD 'cassandra://PigTest/SomeApp' USING CassandraStorage();

为：

rows = LOAD 'cassandra://PigTest/SomeApp?widerows=true' USING CassandraStorage();

我收到以下错误：

java.lang.IndexOutOfBoundsException: Index: 8, Size: 2
    at java.util.ArrayList.rangeCheck(ArrayList.java:604)
    at java.util.ArrayList.get(ArrayList.java:382)
    at org.apache.pig.data.DefaultTuple.get(DefaultTuple.java:156)
    at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject.processInputBag(POProject.java:579)
    at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject.getNext(POProject.java:248)
    at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:316)
    at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:332)
    at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:284)
    at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:290)
    at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:233)
    at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:290)
    at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPreCombinerLocalRearrange.getNext(POPreCombinerLocalRearrange.java:126)
    at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:290)
    at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:233)
    at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:290)
    at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:256)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:271)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:266)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)                                                                                                                                                                                                                                                                                                           
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)

在本地和mapreduce模式下运行时，或者如果我设置PIG_WIDEROW_INPUT = true，就会发生这种情况。

以下Pig Latin脚本将失败并显示“widerows = true”参数。

rows = LOAD 'cassandra://PigTest/SomeApp?widerows=true' USING CassandraStorage();
cols = FOREACH rows GENERATE flatten(columns.name);
DUMP cols;

我似乎无法超越这一点，在使用宽行输入时，不会读取SomeApp列族中的静态列。其他列族也存在同样的问题。

Answer 1

我有类似的问题。这可能是因为get_paged_slices中的错误在以后的1.1.x版本中得到修复。解决方案是将Cassandra升级到 ~~1.1.8~~ 1.1.9

请参阅：

CASSANDRA-4919：StorageProxy.getRangeSlice有时会返回错误的列数
CASSANDRA-4816：已损坏的get_paged_slice
CASSANDRA-5098：CassandraStorage无法以宽幅模式解码名称

Cassandra Pig示例失败，启用了宽行输入

1 个答案: