Question

我正在使用AWS Data Pipeline中的Load S3 data into RDS MySql table模板将csv从S3存储桶导入到我们的RDS MySql中。但是，我（作为具有完全管理员权限的IAM用户）遇到了无法解决的警告：

对象：Ec2Instance - 警告：无法验证S3访问用于作用。请确保角色（ 'DataPipelineDefaultRole'）具有S3：GET *，S3：列出*，S3：将*和STS：AssumeRole权限DataPipeline

Google告诉我不要对DataPipelineDefaultRole和DataPipelineDefaultResourceRole使用默认策略。根据{{3}}和IAM Roles for AWS Data Pipeline的文档，我使用了内联策略并编辑了两个角色的信任关系。

政策DataPipelineDefaultRole：

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:*",
                "datapipeline:DescribeObjects",
                "datapipeline:EvaluateExpression",
                "dynamodb:BatchGetItem",
                "dynamodb:DescribeTable",
                "dynamodb:GetItem",
                "dynamodb:Query",
                "dynamodb:Scan",
                "dynamodb:UpdateTable",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:CancelSpotInstanceRequests",
                "ec2:CreateSecurityGroup",
                "ec2:CreateTags",
                "ec2:DeleteTags",
                "ec2:Describe*",
                "ec2:ModifyImageAttribute",
                "ec2:ModifyInstanceAttribute",
                "ec2:RequestSpotInstances",
                "ec2:RunInstances",
                "ec2:StartInstances",
                "ec2:StopInstances",
                "ec2:TerminateInstances",
                "ec2:AuthorizeSecurityGroupEgress",
                "ec2:DeleteSecurityGroup",
                "ec2:RevokeSecurityGroupEgress",
                "ec2:DescribeNetworkInterfaces",
                "ec2:CreateNetworkInterface",
                "ec2:DeleteNetworkInterface",
                "ec2:DetachNetworkInterface",
                "elasticmapreduce:*",
                "iam:GetInstanceProfile",
                "iam:GetRole",
                "iam:GetRolePolicy",
                "iam:ListAttachedRolePolicies",
                "iam:ListRolePolicies",
                "iam:ListInstanceProfiles",
                "iam:PassRole",
                "rds:DescribeDBInstances",
                "rds:DescribeDBSecurityGroups",
                "redshift:DescribeClusters",
                "redshift:DescribeClusterSecurityGroups",
                "s3:CreateBucket",
                "s3:DeleteObject",
                "s3:Get*",
                "s3:List*",
                "s3:Put*",
                "sdb:BatchPutAttributes",
                "sdb:Select*",
                "sns:GetTopicAttributes",
                "sns:ListTopics",
                "sns:Publish",
                "sns:Subscribe",
                "sns:Unsubscribe",
                "sqs:CreateQueue",
                "sqs:Delete*",
                "sqs:GetQueue*",
                "sqs:PurgeQueue",
                "sqs:ReceiveMessage"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "iam:CreateServiceLinkedRole",
            "Resource": "*",
            "Condition": {
                "StringLike": {
                    "iam:AWSServiceName": [
                        "elasticmapreduce.amazonaws.com",
                        "spot.amazonaws.com"
                    ]
                }
            }
        }
    ]
}

信任关系DataPipelineDefaultRole：

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "ec2.amazonaws.com",
          "elasticmapreduce.amazonaws.com",
          "datapipeline.amazonaws.com"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

政策DataPipelineDefaultResourceRole：

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:*",
                "datapipeline:*",
                "dynamodb:*",
                "ec2:Describe*",
                "elasticmapreduce:AddJobFlowSteps",
                "elasticmapreduce:Describe*",
                "elasticmapreduce:ListInstance*",
                "rds:Describe*",
                "redshift:DescribeClusters",
                "redshift:DescribeClusterSecurityGroups",
                "s3:*",
                "sdb:*",
                "sns:*",
                "sqs:*"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

信任关系DataPipelineDefaultResourceRole：

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

我尝试了几种选项/组合，但警告仍然存在。有谁知道如何解决此权限问题？

Answer 1

我没有看到有关您的政策和角色定义的任何问题。一切看起来不错。我唯一想到的就是定义角色后创建管道的速度有多快？

请记住， IAM策略是全局的，而数据管道存在于特定区域中，因此在创建策略/角色和创建数据管道，AWS需要花费时间来复制所有区域中的IAM策略更改。

Ex. if you are using bash aws-cli to create/update role & then create/activate data-pipeline, insert `sleep Xs` between role & datapipeline creation.

Nitpick ，您无需为ec2.amazonaws.com建立信任关系中的DataPipelineDefaultRole。

Answer 2

回答这个问题可能有点晚了，但我发现您看到的警告消息可能会误导您。如果将管道配置为将日志放入S3存储桶，则仅指定存储桶的根目录而不是路径将显示警告。例如，如果我将配置字段“ Pipeline Log Uri”（在默认配置中找到）设置为s3://bucket-name/，则会看到警告。另一方面，如果我指定一个路径，例如s3://bucket-name/logs，则警告消失。

AWS论坛中的以下线程对于解决这一问题确实很有帮助：https://forums.aws.amazon.com/thread.jspa?threadID=164635。

AWS数据管道：具有针对IAM角色的S3访问权限的问题

2 个答案: