Question

我是Pig的新手，我正在尝试在我们的5节点Hadoop集群上运行以下PigScript。以下脚本为我提供了关系中两列的集合交集

register '/home/workspace/Pig/setIntersecUdf.jar'; 
define Inter com.cs.pig.SetIntersection(); 
a = load '/home/pig/pig-0.12.0/input/location.txt' as (location:chararray); 
b = load '/home/pig/pig-0.12.0/input/location.txt' as (location:chararray); 
c = CROSS a,b parallel 10; 
c = DISTINCT c; 
d = Foreach c generate $0,$1,Inter($0,$1) as intersection; 
e = Filter d by intersection !='[]' parallel 10; 
e = Filter e by $0!=$1 parallel 10; 
Store e into '/home/Documents/Pig_output';

当我运行此脚本而不提供'PARALLEL'选项时，它运行正常。但是，当我添加此选项时，我在reducers的tasktracker堆栈中收到以下错误，并且作业失败。

Java.io.IOException: org.apache.hadoop.ipc.RemoteException: 
java.io.IOException: File /tmp/temp-10926921/tmp-1823693600/_temporary/_attempt_201401171541_0001_r_000000_0/part-r-00000 could only be replicated to 0 nodes, instead of 1 at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639) at 
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736) at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source) at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at 
org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at 
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) at 
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) at 
java.security.AccessController.doPrivileged(Native Method) at 
javax.security.auth.Subject.doAs(Subject.java:396) at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:469) at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:432) at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:404) at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:256) at 
org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176) at 
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:650) at 
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418) at 
org.apache.hadoop.mapred.Child$4.run(Child.java:255) at 
java.security.AccessController.doPrivileged(Native Method) at 
javax.security.auth.Subject.doAs(Subject.java:396) at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.mapred.Child.main(Child.java:249) Caused by: 
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /tmp/temp-10926921/tmp-1823693600/_temporary/_attempt_201401171541_0001_r_000000_0/part-r-00000 could only be replicated to 0 nodes, instead of 1 at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639) at 
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736) at 
sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source) at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at 
org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at 
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) at 
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) at 
java.security.AccessController.doPrivileged(Native Method) at 
javax.security.auth.Subject.doAs(Subject.java:396) at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) at 
org.apache.hadoop.ipc.Client.call(Client.java:1107) at 
org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at 
$Proxy2.addBlock(Unknown Source) at 
sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85) at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62) at $Proxy2.addBlock(Unknown Source) at 
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686) at 
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546) at 
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749) at 
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989)

此外，失败作业的诊断信息如下：

   # of failed Reduce Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201401171541_0001_r_000002

请帮助！

Answer 1

问题似乎是您的reducer上的磁盘空间不足。 CROSS操作非常昂贵，因为它会写出所有可能的记录对。您有321,372条记录，因此当您使用CROSS时，您的数据将会大幅增加321,372。

由于您的数据大约为6 MB，因此需要写入大约1.9 TB的数据。当您指定PARALLEL 10时，它会分布在10个减速器上，这可能不会超出任何单个机器的磁盘空间。但是如果没有PARALLEL 10，Pig会默认为单个reducer（可能是因为输入的小尺寸）并且尝试在该单个机器上写入所有数据。

也许你可以在FILTER之后立即积极CROSS你的结果，这样Pig会在尝试写入磁盘之前抛出大部分记录。或者您需要重新考虑数据流，这样就不需要使用CROSS。

添加并行选项会使Pig中的作业失败

1 个答案: