Pig Cassandra集群ClassNotFoundException：org.apache.cassandra.hadoop.ColumnFamilySplit

Question

我正在尝试运行Cassandra-0.8.5，Hadoop 0.2.0和Pig 0.8.1。我运行一个非常简单的猪脚本

rows = LOAD 'cassandra://pygmalion/$CF' USING CassandraStorage() AS (key, columns: bag {T: tuple(name, value)});
counted = foreach (group rows all) generate COUNT($1);
dump counted;

如果我运行本地模式，它运行良好。但是，如果我运行mapreduce模式，我总是得到如下错误信息，我没有想法。任何帮助或后悔将不胜感激。 pig_813399.log：

后端错误消息

java.io.IOException: java.lang.ClassNotFoundException: org.apache.cassandra.hadoop.ColumnFamilySplit
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSplit.readFields(PigSplit.java:225)
        at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
        at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
        at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:349)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:611)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:416)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
        at org.apache.hadoop.mapred.Child.main(Child.java:264) Caused by: java.lang.ClassNotFoundException: org.apache.c

猪堆痕迹

ERROR 2997: Unable to recreate exception from backed error: java.io.IOException: java.lang.ClassNotFoundException: org.apache.cassandra.hadoop.ColumnFamilySplit

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias counted. Backend error : Unable to recreate exception from backed error: java.io.IOException: java.lang.ClassNotFoundException: org.apache.cassandra.hadoop.ColumnFamilySplit
        at org.apache.pig.PigServer.openIterator(PigServer.java:753)
        at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:615)
        at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
        at org.apache.pig.tools.grunt.GruntParser.loadScript(GruntParser.java:477)
        at org.apache.pig.tools.grunt.GruntParser.processScript(GruntParser.java:422)
        at org.apache.pig.tools.pigscript.parser.PigScriptParser.Script(PigScriptParser.java:692)
        at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:425)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:144)
        at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
        at org.apache.pig.Main.run(Main.java:455)
        at org.apache.pig.Main.main(Main.java:107)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2997: Unable to recreate exception from backed error: java.io.IOExcepti
on: java.lang.ClassNotFoundException: org.apache.cassandra.hadoop.ColumnFamilySplit
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:221)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:151)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:337)
        at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:382)
        at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1209)
        at org.apache.pig.PigServer.storeEx(PigServer.java:885)
        at org.apache.pig.PigServer.store(PigServer.java:827)
        at org.apache.pig.PigServer.openIterator(PigServer.java:739)

Answer 1

听起来您需要更新HADOOP_CLASSPATH环境变量以包含cassandra jar。你需要重新启动hadoop进程。

export HADOOP_CLASSPATH=/path/to/cassandra/lib/*:$HADOOP_CLASSPATH

有关运行hadoop / pig对抗cassandra的其他可能问题，请参阅http://wiki.apache.org/cassandra/HadoopSupport。