我一直在尝试使用Pig将数据加载到HBase表中。我添加了引用HBase库的Pig类路径,并尝试在本地模式下运行Pig脚本。我首先使用'info'列系列在HBase中创建了一个表'weather'。
引用HBase库的Pig类路径是:
export PIG_CLASSPATH=$PIG_INSTALL/pig-0.13.0-h1.jar:$HBASE_HOME/lib/*:$HADOOP_COMMON_LIB_NATIVE_DIR/lib/*:$PIG_CLASSPATH
猪脚本如下:
data = LOAD '/home/uzair/200707hourly.txt' USING PigStorage(',');
temp_info = FOREACH data GENERATE $1, $10;
rank = RANK temp_info;
final = FILTER rank BY $0 IS NOT NULL;
STORE final INTO 'hbase://weather' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('info:date info:temp');
当我使用pig -x local -f /home/uzair/pigload.pig运行此脚本时,出现以下错误:
14/09/25 14:05:59 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
14/09/25 14:05:59 INFO pig.ExecTypeProvider: Picked LOCAL as the ExecType
2014-09-25 14:06:00,247 [main] INFO org.apache.pig.Main - Apache Pig version 0.13.0 (r1606446) compiled Jun 29 2014, 02:29:34
2014-09-25 14:06:00,253 [main] INFO org.apache.pig.Main - Logging error messages to: /home/uzair/pig_1411634160241.log
2014-09-25 14:06:04,061 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/uzair/.pigbootup not found
2014-09-25 14:06:04,444 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2014-09-25 14:06:08,717 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
2014-09-25 14:06:08,718 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=ubuntu
2014-09-25 14:06:08,718 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.7.0_45
2014-09-25 14:06:08,718 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
2014-09-25 14:06:08,719 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/local/java/jdk1.7.0_45/jre
2014-09-25 14:06:08,719 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/home/uzair/hadoop/hadoop-1.2.1/libexec/../conf:/usr/local/java/jdk1.7.0_45/lib/tools.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/..:/home/uzair/hadoop/hadoop-1.2.1/libexec/../hadoop-core-1.2.1.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/asm-3.2.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/aspectjrt-1.6.11.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/aspectjtools-1.6.11.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/commons-beanutils-1.7.0.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/commons-beanutils-core-1.8.0.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/commons-cli-1.2.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/commons-codec-1.4.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/commons-collections-3.2.1.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/commons-configuration-1.6.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/commons-daemon-1.0.1.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/commons-digester-1.8.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/commons-el-1.0.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/commons-httpclient-3.0.1.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/commons-io-2.1.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/commons-lang-2.4.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/commons-logging-1.1.1.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/commons-logging-api-1.0.4.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/commons-math-2.1.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/commons-net-3.1.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/core-3.1.1.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/hadoop-capacity-scheduler-1.2.1.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/hadoop-fairscheduler-1.2.1.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/hadoop-thriftfs-1.2.1.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/hsqldb-1.8.0.10.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/jackson-core-asl-1.8.8.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/jasper-compiler-5.5.12.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/jasper-runtime-5.5.12.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/jdeb-0.8.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/jersey-core-1.8.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/jersey-json-1.8.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/jersey-server-1.8.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/jets3t-0.6.1.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/jetty-6.1.26.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/jetty-util-6.1.26.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/jsch-0.1.42.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/junit-4.5.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/kfs-0.2.2.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/log4j-1.2.15.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/mockito-all-1.8.5.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/oro-2.0.8.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/servlet-api-2.5-20081211.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/slf4j-api-1.4.3.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/slf4j-log4j12-1.4.3.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/xmlenc-0.52.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/jsp-2.1/jsp-2.1.jar:/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/usr/local/pig-0.13.0/bin/../conf:/usr/local/java/jdk1.7.0_45/lib/tools.jar:/usr/local/pig-0.13.0/pig-0.13.0-h1.jar:/lib/*:/home/uzair/hadoop/hadoop-1.2.1/lib/native/lib/*:/usr/local/pig-0.13.0/pig-0.13.0-h1.jar:/lib/*:/home/uzair/hadoop/hadoop-1.2.1/lib/native/lib/*::/usr/local/pig-0.13.0/bin/../lib/accumulo-core-1.5.0.jar:/usr/local/pig-0.13.0/bin/../lib/accumulo-fate-1.5.0.jar:/usr/local/pig-0.13.0/bin/../lib/accumulo-server-1.5.0.jar:/usr/local/pig-0.13.0/bin/../lib/accumulo-start-1.5.0.jar:/usr/local/pig-0.13.0/bin/../lib/accumulo-trace-1.5.0.jar:/usr/local/pig-0.13.0/bin/../lib/avro-1.7.5.jar:/usr/local/pig-0.13.0/bin/../lib/avro-mapred-1.7.5.jar:/usr/local/pig-0.13.0/bin/../lib/avro-tools-1.7.5-nodeps.jar:/usr/local/pig-0.13.0/bin/../lib/groovy-all-1.8.6.jar:/usr/local/pig-0.13.0/bin/../lib/hbase-0.94.1.jar:/usr/local/pig-0.13.0/bin/../lib/jruby-complete-1.6.7.jar:/usr/local/pig-0.13.0/bin/../lib/js-1.7R2.jar:/usr/local/pig-0.13.0/bin/../lib/json-simple-1.1.jar:/usr/local/pig-0.13.0/bin/../lib/jython-standalone-2.5.3.jar:/usr/local/pig-0.13.0/bin/../lib/piggybank.jar:/usr/local/pig-0.13.0/bin/../lib/protobuf-java-2.4.0a.jar:/usr/local/pig-0.13.0/bin/../lib/zookeeper-3.4.5.jar:/usr/local/pig-0.13.0/bin/../pig-0.13.0-withouthadoop-h1.jar:
2014-09-25 14:06:08,720 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/home/uzair/hadoop/hadoop-1.2.1/libexec/../lib/native/Linux-amd64-64
2014-09-25 14:06:08,720 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2014-09-25 14:06:08,720 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2014-09-25 14:06:08,720 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2014-09-25 14:06:08,720 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2014-09-25 14:06:08,721 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.13.0-36-generic
2014-09-25 14:06:08,721 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=uzair
2014-09-25 14:06:08,721 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/uzair
2014-09-25 14:06:08,721 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/uzair
2014-09-25 14:06:08,724 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2181 sessionTimeout=180000 watcher=hconnection
2014-09-25 14:06:08,940 [main] INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - The identifier of this process is 5252@ubuntu
2014-09-25 14:06:08,958 [main-SendThread(localhost:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2014-09-25 14:06:09,040 [main-SendThread(localhost:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2181, initiating session
2014-09-25 14:06:09,091 [main-SendThread(localhost:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x148abe93714000d, negotiated timeout = 90000
2014-09-25 14:16:49,174 [main] ERROR org.apache.hadoop.hbase.mapreduce.TableOutputFormat - org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for weather,,99999999999999 after 10 tries.
2014-09-25 14:16:49,238 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2999: Unexpected internal error. org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for weather,,99999999999999 after 10 tries.
Details at logfile: /home/uzair/pig_1411634160241.log
[编辑] Pig堆栈跟踪:
Pig Stack Trace
---------------
ERROR 2999: Unexpected internal error. org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for weather,,99999999999999 after 10 tries.
java.lang.RuntimeException: org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for weather,,99999999999999 after 10 tries.
at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:206)
at org.apache.pig.backend.hadoop.hbase.HBaseStorage.getOutputFormat(HBaseStorage.java:890)
at org.apache.pig.newplan.logical.visitor.InputOutputFileValidatorVisitor.visit(InputOutputFileValidatorVisitor.java:69)
at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)
at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
at org.apache.pig.newplan.logical.relational.LogicalPlan.validate(LogicalPlan.java:210)
at org.apache.pig.PigServer$Graph.compile(PigServer.java:1744)
at org.apache.pig.PigServer$Graph.access$300(PigServer.java:1420)
at org.apache.pig.PigServer.execute(PigServer.java:1333)
at org.apache.pig.PigServer.executeBatch(PigServer.java:392)
at org.apache.pig.PigServer.executeBatch(PigServer.java:375)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:170)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:232)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:203)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
at org.apache.pig.Main.run(Main.java:479)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
Caused by: org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for weather,,99999999999999 after 10 tries.
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:936)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:845)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:172)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:131)
at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:201)
... 27 more
================================================================================
使用HBase管理的Zookeeper,HBase安装处于伪分发模式。我不知道我在这里失踪了什么。在这方面的任何帮助将受到高度赞赏。如果您需要更多信息,如主服务器或区域服务器日志,请与我们联系。
由于 Uzair Syed