Question

我正在尝试将数据从HDFS加载到Pig但我收到错误，因为创建数据存储失败。我执行的命令是：

movies = LOAD 'hdfs://localhost:9000/Movie_Lens/ratings' USING PigStorage(':') AS (user_id, dummy1, movie_id, dummy2, movie_rating, dummy3, timestamp);

我试图在堆栈溢出中找到所提到的问题，但我得到的链接与HDFS和Pig无关，它们与HDFS和HBase或Pig和HBase有关。

日志文件的详细信息如下所述。

在日志文件的某处我发现了这一点：引起：org.apache.hadoop.ipc.RemoteException：服务器IPC版本9无法与客户端版本4通信

猪堆痕迹

错误1200：无法创建DataStorage

Failed to parse: Failed to create DataStorage
    at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:201)
    at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1707)
    at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1680)
    at org.apache.pig.PigServer.registerQuery(PigServer.java:623)
    at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:1082)
    at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:505)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)
    at org.apache.pig.Main.run(Main.java:565)
    at org.apache.pig.Main.main(Main.java:177)
Caused by: java.lang.RuntimeException: Failed to create DataStorage
    at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
    at org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:53)
    at org.apache.pig.builtin.JsonMetadata.findMetaFile(JsonMetadata.java:109)
    at org.apache.pig.builtin.JsonMetadata.getSchema(JsonMetadata.java:189)
    at org.apache.pig.builtin.PigStorage.getSchema(PigStorage.java:538)
    at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:175)
    at org.apache.pig.newplan.logical.relational.LOLoad.<init>(LOLoad.java:89)
    at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:901)
    at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568)
    at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625)
    at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
    at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
    at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
    at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191)
    ... 10 more
Caused by: org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
    at org.apache.hadoop.ipc.Client.call(Client.java:1070)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
    at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
    at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
    at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:70)
    ... 23 more

为了解决这个问题，我试着去做蚂蚁＆＃39; 所以当我运行命令

时

bash ant -version

在ant bin文件夹中它正在工作但是当我运行命令时

bash ant clean jar-all -Dhadoopversion=23

在bin文件夹中

它无法正常工作。在一些链接中，我发现新版本的pig没有jar-all命令，所以我尝试了以下命令

bash ant clean jar -Dhadoopversion=23

并且此命令也无效。

获取错误尝试使用MovieLens数据从HDFS加载数据时无法创建数据存储

猪堆痕迹

0 个答案: