Question

我正在尝试使用hdfs文件中的一些信息加入hbase表。我对hbase和猪比较新。

set hbase.zookeeper.quorum 'mprhdp2-dev.local' 
set hbase.zookeeper.property.clientPort 5100
A = LOAD '/raw/JoinTest2.txt' USING PigStorage('\t') AS (Id:chararray, text:chararray);   
B = LOAD 'hbase://MediaInformationMock' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('MediaInformation:UmbrellaCode', '-loadKey    true -caster=HBaseBinaryConverter') AS (Id:chararray, UmbrellaCode:chararray);

J = JOIN B BY $0, A BY $0;  
DUMP J;


2012-10-02 08:23:59,923 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1003: Unable to find an operator for alias J
Details at logfile: /home/user1/pig_1349183747159.log

以下是日志文件中的内容：

Pig Stack Trace
---------------
ERROR 2017: Internal error creating job configuration.

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias J
    at org.apache.pig.PigServer.openIterator(PigServer.java:857)
    at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:682)
    at     org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
    at org.apache.pig.Main.run(Main.java:490)
    at org.apache.pig.Main.main(Main.java:111)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias J
    at org.apache.pig.PigServer.storeEx(PigServer.java:956)
    at org.apache.pig.PigServer.store(PigServer.java:919)
    at org.apache.pig.PigServer.openIterator(PigServer.java:832)
    ... 12 more
Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: ERROR 2017: Internal error creating job configuration.
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:739)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:259)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:180)
    at org.apache.pig.PigServer.launchPlan(PigServer.java:1270)
    at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1255)
    at org.apache.pig.PigServer.storeEx(PigServer.java:952)
    ... 14 more
Caused by: java.io.IOException: No FileSystem for scheme: hbase
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1416)
    at org.apache.hadoop.fs.FileSystem.access$100(FileSystem.java:69)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1453)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1435)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:232)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:191)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getTotalInputFileSize(JobControlCompiler.java:788)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.estimateNumberOfReducers(JobControlCompiler.java:762)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:600)
    ... 19 more

有什么想法吗？

猪版本0.10.0 HBase 0.92.1 Hadoop 0.20.2-dev

DUMP A:
(1,A)
(2,B)
(3,C)
(4,D)
(5,E)

DUMP B:
(1,S)
(10,C)
(11,D)
(12,D)
(13,M)
(14,N)
(15,W)
(16,I)
(17,I)
(18,R)
(19,T)
(2,S)
(20,D)
(21,I)
(22,S)
(23,S)
(24,S)
(25,S)
(3,S)
(4,S)
(5,S)
(6,S)
(7,R)
(8,S)
(9,T)

Answer 1

我怀疑您hbase-site.xml中没有PIG_CLASSPATH。

尝试将它与zookeeper，guava和hbase jar一起添加。用

开始养猪

pig --conf $HBASE_HOME/conf my.pig

并在脚本中注册jar。

在pig错误中加入hbase表：没有用于scheme的文件系统：hbase

1 个答案: