如何在aws数据管道中设置多个s3桶的前提条件?

时间:2017-10-10 17:37:03

标签: amazon-s3 amazon-data-pipeline

如何在数据管道中设置先决条件,以便检查s3存储桶是否存在,如果不存在则为FAILS?我可以轻松地设置1个桶,但我无法弄清楚如何更新管道以检查多个S3keys。

这是我到目前为止所拥有的:

{
  "objects": [
...
    {
      "stage": "true",
      "name": "ShellCommandActivityObj",
      "id": "ShellCommandActivityObj",
      "scriptArgument": [
        "#{myEfsSource}",
        "#{myInterval}",
        "#{myRetainedBackups}",
        "#{myEfsID}"
      ],
      "runsOn": {
        "ref": "EC2ResourceObj"
      },
      "type": "ShellCommandActivity",
      "command": "#{myShellCmd}",
      "onSuccess": {
         "ref": "Notify_on_success"
      },
      "onFail": {
         "ref": "Notify_on_Failure"
      },
      "precondition": {
         "ref": "DestinationBucketStatus",
         "ref": "BucketWithScriptStatus"
      }
    }
    ....
}

我尝试在前置条件中使用list但后来我收到以下错误:

Invalid type for parameter pipelineObjects[4].fields[3].refValue, value: [u'DestinationBucketStatus', u'BucketWithScriptStatus'], type: <type 'list'>, valid types: <type 'basestring'>

任何建议表示赞赏, 谢谢

1 个答案:

答案 0 :(得分:0)

万一有人无法弄明白。

代码发布在AWS论坛上:

https://forums.aws.amazon.com/thread.jspa?threadID=266162