Question

我正在尝试使用在Jenkins作业中通过CLI从我正在构建的EMR集群复制的现有安全设置在VPC内创建EMR集群。

我无法使它正常工作。如果我未指定ec2-attributes，它将失败（并尝试在我不需要的VPC之外创建集群）。如果我包括他们，它将失败。以这种方式尝试启动集群（因此我认为IAM没问题）：

The field "Ec2InstanceAttributes" is not supported by Step Functions

这是我要使用的配置：

"Create_A_Cluster": {

  "Type": "Task",
  "Resource": "arn:aws:states:::elasticmapreduce:createCluster.sync",
  "Parameters": {
    "Name": "WorkflowCluster",
    "VisibleToAllUsers": true,
    "ReleaseLabel": "emr-6.0.0",
    "Applications": [
      {
        "Name": "Hive"
      }
    ],
    "Ec2InstanceAttributes": {
        "Ec2KeyName": "URMOM",
        "Ec2SubnetId": "subnet-4543b1f",
        "RequestedEc2SubnetIds": [
            "subnet-1235cb1f"
        ],
        "Ec2AvailabilityZone": "us-east-1a",
        "RequestedEc2AvailabilityZones": [],
        "IamInstanceProfile": "emr-ec2-role-sexy-dev",
        "EmrManagedMasterSecurityGroup": "sg-669e32158412f98b6",
        "EmrManagedSlaveSecurityGroup": "sg-2169f6e0142047ed1",
        "ServiceAccessSecurityGroup": "sg-04d4a4db4cac4c1d7",
        "AdditionalMasterSecurityGroups": [],
        "AdditionalSlaveSecurityGroups": []
    },
    "ServiceRole": "emr-service-role-sexy-dev",
    "JobFlowRole": "emr-ec2-role-sexy-dev",
    "LogUri": "s3n://emr-logs-sexy-dev/logs/sexy/dev/",
    "Instances": {
      "KeepJobFlowAliveWhenNoSteps": true,
      "InstanceFleets": [
        {
          "InstanceFleetType": "CORE",
          "TargetOnDemandCapacity": 2,
          "TargetSpotCapacity": 0,
          "LaunchSpecifications": {},
          "InstanceTypeConfigs": [
            {
              "WeightedCapacity": 4,
              "EbsConfiguration": {
                "EbsBlockDeviceConfigs": [
                  {
                    "VolumeSpecification": {
                      "SizeInGB": 32,
                      "VolumeType": "gp2"
                    },
                    "VolumesPerInstance": 2
                  }
                ]
              },
              "BidPriceAsPercentageOfOnDemandPrice": 100,
              "InstanceType": "m5.xlarge"
            }
          ],
          "Name": "Core - 2"
        },
        {
          "InstanceFleetType": "MASTER",
          "TargetOnDemandCapacity": 1,
          "TargetSpotCapacity": 0,
          "LaunchSpecifications": {},
          "InstanceTypeConfigs": [
            {
              "WeightedCapacity": 1,
              "EbsConfiguration": {
                "EbsBlockDeviceConfigs": [
                  {
                    "VolumeSpecification": {
                      "SizeInGB": 32,
                      "VolumeType": "gp2"
                    },
                    "VolumesPerInstance": 2
                  }
                ]
              },
              "BidPriceAsPercentageOfOnDemandPrice": 100,
              "InstanceType": "m5.xlarge"
            }
          ],
          "Name": "Master - 1"
        }
      ]
    }
  }

Answer 1

没有太多的运行EMR的经验，但是我确实有使用Step功能的经验。您所用的目标API的文档没有“ Ec2InstanceAttributes”作为参数。它的内容有些分散在其他领域，例如实例。

我建议尝试与此处的规范保持一致：https://docs.aws.amazon.com/emr/latest/APIReference/API_RunJobFlow.html

Answer 2

对于可能会通过网络搜索找到这些人的方式，我可以通过将 Ec2SubnetId 指定为 Instances 的子代来实现，并将其内置到VPC中想要的。

使用AWS Step Functions在VPC内创建集群

2 个答案: