AWS数据管道作业失败,但没有错误消息或错误代码

时间:2013-09-03 21:50:25

标签: amazon-web-services amazon-data-pipeline

我尝试运行数据管道作业,但EmrActivity步骤已达到FAILED状态,但没有错误代码或错误消息:

名称: @EMR集群执行work_2013-09-03T16:15:00 查看实例字段 描述: 最新尝试次数:3,尝试左:0 选择此实例的尝试:

状态: 失败 错误代码: 错误讯息:

任何想法为什么?我在哪里可以找到有关潜在问题的更多信息?

工作很简单:启动EMR集群并运行猪脚本(其中xxx是我的存储桶名称):


    {
    "objects": [
    {
    "id":"Default",
    "failureAndRerunMode":"cascade" 
    },
    {
    "id" : "MyScheduleID",
    "type" : "Schedule",
    "period" : "1 hour",
    "startDateTime" : "2013-09-03T19:00:00",
    "endDateTime" : "2013-09-03T20:00:00"
    },
    {
    "id" : "MyEmrCluster",
    "name" : "EMR cluster to perform the work",
    "type" : "EmrCluster",
    "hadoopVersion" : "0.20",
    "masterInstanceType" : "m1.small",
    "coreInstanceType" : "m1.medium",
    "coreInstanceCount" : "2",
    "terminateAfter": "1 Hours",
    "schedule": {
    "ref": "MyScheduleID"
    },
    "logUri":"s3://xxx/amazonlogs",
    "emrLogUri":"s3://xxx/amazonlogs"
    },

    {
    "id" : "MyEmrActivity",
    "name" : "Work to perform on my data",
    "type" : "EmrActivity",
    "runsOn" : {"ref" : "MyEmrCluster"},
    "schedule": {
    "ref": "MyScheduleID"
    },
    "step": "s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar,s3://us-east-1.elasticmapreduce/libs/pig/pig-script,--base-path,s3://us-east-1.elasticmapreduce/libs/pig/,--install-pig,--pig-versions,latest",
    "step": "s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar,s3://us-east-1.elasticmapreduce/libs/pig/pig-script,--base-path,s3://us-east-1.elasticmapreduce/libs/pig/,--pig-versions,latest,--run-pig-script,--args,-f,s3://xxx/carls_minimal_script.pig"
    }

    ]
    }

这个配置看起来不错吗? 我在s3:// xxx / amazonlogs

中看不到任何内容

1 个答案:

答案 0 :(得分:2)

以下是您可以尝试的几件事

  1. 转到“https://console.aws.amazon.com/elasticmapreduce/home”,找到已启动的相应群集(基于时间戳),单击“调试”,您应找到有关每个步骤的日志。

  2. 或者从AWS控制台启动EMR集群,登录主节点,运行Pig脚本以检查其是否正常工作。