Question

我编写的猪脚本是在我的本地运行的4gb数据上但是当我尝试在EMR集群上运行它时，它会在一个特定的组中挂起来声明。这是我得到的错误...

    at org.apache.tez.dag.app.dag.impl.VertexImpl$NoOpVertexManager.onVertexStateUpdated(VertexImpl.java:4528)
    at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEventOnVertexStateUpdate.invoke(VertexManager.java:564)
    at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent$1.run(VertexManager.java:647)
    at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent$1.run(VertexManager.java:642)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
    at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent.call(VertexManager.java:642)
    at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent.call(VertexManager.java:631)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

引起：org.apache.tez.dag.app.dag.impl.AMUserCodeException：无法初始化Edge，EdgeInfo：sourceVertexName = scope-325，destinationVertexName = scope-329 在org.apache.tez.dag.app.dag.impl.Edge.initialize（Edge.java:174）在org.apache.tez.dag.app.dag.impl.Edge.setEdgeProperty(Edge.java:196） at org.apache.tez.dag.app.dag.impl.VertexImpl.setParallelismWrapper（VertexImpl.java:1724） ......还有16个引起：java.lang.IllegalStateException 在com.google.common.base.Preconditions.checkState（Preconditions.java:133）在org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager $ CustomShuffleEdgeManager.initialize（ShuffleVertexManager.java:251）在org.apache.tez.dag.app.dag.impl.Edge.initialize（Edge.java:171） ......还有18个 ] 顶点已杀死，vertexName = scope-346，vertexId = vertex_1509345097826_0006_1_11，diagnostics = [Vertex收到Kill in NEW state。，Vertex vertex_1509345097826_0006_1_11 [scope-346]由于：OTHER_VERTEX_FAILURE导致死亡/失败顶点已杀死，vertexName = scope-344，vertexId = vertex_1509345097826_0006_1_10，诊断= [顶点收到杀死处于新状态。，顶点vertex_1509345097826_0006_1_10 [scope-344]由于以下原因导致死亡/失败：OTHER_VERTEX_FAILURE] 顶点已杀死，vertexName = scope-334，vertexId = vertex_1509345097826_0006_1_09，diagnostics = [顶点收到KILL in INITED状态。，Vertex vertex_1509345097826_0006_1_09 [scope-334]由于：OTHER_VERTEX_FAILURE导致死亡/失败顶点已杀死，vertexName = scope-317，vertexId = vertex_1509345097826_0006_1_02，diagnostics = [顶点收到KILL in INITED状态。，Vertex vertex_1509345097826_0006_1_02 [scope-317]由于以下原因而被杀/失败：OTHER_VERTEX_FAILURE] 顶点已杀死，vertexName = scope-316，vertexId = vertex_1509345097826_0006_1_01，diagnostics = [在RUNNING状态下，Vertex收到Kill。由于OTHER_VERTEX_FAILURE，Vertex未成功，failedTasks：0 killedTasks：1，Vertex vertex_1509345097826_0006_1_01 [scope-316] kill / failed由于：OTHER_VERTEX_FAILURE] 顶点已杀死，vertexName = scope-315，vertexId = vertex_1509345097826_0006_1_00，diagnostics = [Vertunder在RUNNING状态下收到Kill。由于OTHER_VERTEX_FAILURE，Vertex未成功，failTasks：0 killedTasks：68，Vertex vertex_1509345097826_0006_1_00 [scope-315] kill / failed由于：OTHER_VERTEX_FAILURE] 由于VERTEX_FAILURE，DAG未成功。 failedVertices：6 deathVertices：6

Answer 1

你在运行什么猪代码？提供完整的信息以及数据和完整的猪代码？

Pig脚本在group by语句中挂起

1 个答案: