自动缩放蓝绿ECS集群

时间:2018-02-21 23:42:43

标签: autoscaling amazon-ecs

我有一个基于https://aws.amazon.com/blogs/compute/bluegreen-deployments-with-amazon-ecs/在Terraform中构建的蓝绿色ECS群集。那篇文章和相关的Github repo没有提到自动缩放。

我正在努力让Application Autoscaling正常工作,但我没有看到我的EC2实例数量上升或下降。目前我正在使用目标跟踪ECSServiceAverageCPUUtilization,目标值为15.换句话说,我希望ECS在群集中的每个EC2实例上维持15%的平均CPU利用率。如果平均CPU利用率攀升,则自动缩放应该向外扩展以使平均值回落;如果CPU平均值下降,则群集应该缩小。(15%太低,但我试图更容易地进行缩放。)

实际CPU利用率高于50%,因此不会进行扩展。我怀疑主动(蓝色)和非主动(绿色)服务相互决斗。

如何在ECS上使用Blue-Green群集获得Autoscaling,Application或Classic?我喜欢目标跟踪的想法,但我对其他想法持开放态度。

扩展政策:

$ aws application-autoscaling describe-scaling-policies --service-namespace ecs \
    --policy-names hub-prod_blue_cpu_target_tracking hub-prod_green_cpu_target_tracking
{
    "ScalingPolicies": [
        {
            "PolicyName": "hub-prod_blue_cpu_target_tracking",
            "ScalableDimension": "ecs:service:DesiredCount",
            "ResourceId": "service/hub-prod_ecs_cluster/hub-prod_es_blue",
            "CreationTime": 1519243876.638,
            "PolicyARN": "arn:aws:autoscaling:us-east-1:ORGID:scalingPolicy:db09a76b-f260-4699-b5aa-cb837086b87d:resource/ecs/service/hub-prod_ecs_cluster/hub-prod_es_blue:policyName/hub-prod_blue_cpu_target_tracking",
            "PolicyType": "TargetTrackingScaling",
            "TargetTrackingScalingPolicyConfiguration": {
                "TargetValue": 15.0,
                "PredefinedMetricSpecification": {
                    "PredefinedMetricType": "ECSServiceAverageCPUUtilization"
                },
                "ScaleOutCooldown": 300,
                "DisableScaleIn": false,
                "ScaleInCooldown": 300
            },
            "Alarms": [
                {
                    "AlarmName": "TargetTracking-service/hub-prod_ecs_cluster/hub-prod_es_blue-AlarmHigh-768f842f-5811-4856-bccf-d73b91f20bfb",
                    "AlarmARN": "arn:aws:cloudwatch:us-east-1:ORGID:alarm:TargetTracking-service/hub-prod_ecs_cluster/hub-prod_es_blue-AlarmHigh-768f842f-5811-4856-bccf-d73b91f20bfb"
                },
                {
                    "AlarmName": "TargetTracking-service/hub-prod_ecs_cluster/hub-prod_es_blue-AlarmLow-d710adc7-5414-46a5-8222-fde9d816f4ee",
                    "AlarmARN": "arn:aws:cloudwatch:us-east-1:ORGID:alarm:TargetTracking-service/hub-prod_ecs_cluster/hub-prod_es_blue-AlarmLow-d710adc7-5414-46a5-8222-fde9d816f4ee"
                }
            ],
            "ServiceNamespace": "ecs"
        },
        {
            "PolicyName": "hub-prod_green_cpu_target_tracking",
            "ScalableDimension": "ecs:service:DesiredCount",
            "ResourceId": "service/hub-prod_ecs_cluster/hub-prod_es_green",
            "CreationTime": 1519243876.577,
            "PolicyARN": "arn:aws:autoscaling:us-east-1:ORGID:scalingPolicy:a407d322-c2a1-4363-9be9-1833f6d3dfce:resource/ecs/service/hub-prod_ecs_cluster/hub-prod_es_green:policyName/hub-prod_green_cpu_target_tracking",
            "PolicyType": "TargetTrackingScaling",
            "TargetTrackingScalingPolicyConfiguration": {
                "TargetValue": 15.0,
                "PredefinedMetricSpecification": {
                    "PredefinedMetricType": "ECSServiceAverageCPUUtilization"
                },
                "ScaleOutCooldown": 300,
                "DisableScaleIn": false,
                "ScaleInCooldown": 300
            },
            "Alarms": [
                {
                    "AlarmName": "TargetTracking-service/hub-prod_ecs_cluster/hub-prod_es_green-AlarmHigh-d7a722a6-b7ab-4bad-846f-d728d4c03763",
                    "AlarmARN": "arn:aws:cloudwatch:us-east-1:ORGID:alarm:TargetTracking-service/hub-prod_ecs_cluster/hub-prod_es_green-AlarmHigh-d7a722a6-b7ab-4bad-846f-d728d4c03763"
                },
                {
                    "AlarmName": "TargetTracking-service/hub-prod_ecs_cluster/hub-prod_es_green-AlarmLow-074e9dde-7936-4b53-ba31-0665190c084f",
                    "AlarmARN": "arn:aws:cloudwatch:us-east-1:ORGID:alarm:TargetTracking-service/hub-prod_ecs_cluster/hub-prod_es_green-AlarmLow-074e9dde-7936-4b53-ba31-0665190c084f"
                }
            ],
            "ServiceNamespace": "ecs"
        }
    ]
}

可扩展目标:

$ aws application-autoscaling describe-scalable-targets --service-namespace ecs \
    --resource-ids service/hub-prod_ecs_cluster/hub-prod_es_blue service/hub-prod_ecs_cluster/hub-prod_es_green
{
    "ScalableTargets": [
        {
            "ScalableDimension": "ecs:service:DesiredCount",
            "ResourceId": "service/hub-prod_ecs_cluster/hub-prod_es_blue",
            "RoleARN": "arn:aws:iam::ORGID:role/aws-service-role/ecs.application-autoscaling.amazonaws.com/AWSServiceRoleForApplicationAutoScaling_ECSService",
            "CreationTime": 1519240496.0,
            "MinCapacity": 3,
            "ServiceNamespace": "ecs",
            "MaxCapacity": 30
        },
        {
            "ScalableDimension": "ecs:service:DesiredCount",
            "ResourceId": "service/hub-prod_ecs_cluster/hub-prod_es_green",
            "RoleARN": "arn:aws:iam::ORGID:role/aws-service-role/ecs.application-autoscaling.amazonaws.com/AWSServiceRoleForApplicationAutoScaling_ECSService",
            "CreationTime": 1519167581.233,
            "MinCapacity": 3,
            "ServiceNamespace": "ecs",
            "MaxCapacity": 30
        }
    ]
}

扩展活动。 (1519252380 = 2018-02-22T2230Z,就在我将ALB Listener的默认目标组修改为蓝色之前。)

蓝:

$ aws application-autoscaling describe-scaling-activities --service-namespace ecs \
    --resource-id service/hub-prod_ecs_cluster/hub-prod_es_blue \
    | jq '.ScalingActivities | .[] | select(.StartTime >= 1519252380)'
{
"ScalableDimension": "ecs:service:DesiredCount",
"Description": "Setting desired count to 24.",
"ResourceId": "service/hub-prod_ecs_cluster/hub-prod_es_blue",
"ActivityId": "db73de6d-3918-4f24-94b7-de7437c767e4",
"StartTime": 1519252897.326,
"ServiceNamespace": "ecs",
"Cause": "monitor alarm TargetTracking-service/hub-prod_ecs_cluster/hub-prod_es_blue-AlarmHigh-768f842f-5811-4856-bccf-d73b91f20bfb in state ALARM triggered policy hub-prod_blue_cpu_target_tracking",
"StatusMessage": "Successfully set desired count to 24. Waiting for change to be fulfilled by ecs.",
"StatusCode": "InProgress"
}
{
"ScalableDimension": "ecs:service:DesiredCount",
"Description": "Setting desired count to 21.",
"ResourceId": "service/hub-prod_ecs_cluster/hub-prod_es_blue",
"ActivityId": "dce5efc9-2c93-4799-87bb-0d344a917de8",
"StartTime": 1519252837.315,
"ServiceNamespace": "ecs",
"EndTime": 1519252907.177,
"Cause": "monitor alarm TargetTracking-service/hub-prod_ecs_cluster/hub-prod_es_blue-AlarmHigh-768f842f-5811-4856-bccf-d73b91f20bfb in state ALARM triggered policy hub-prod_blue_cpu_target_tracking",
"StatusMessage": "Successfully set desired count to 21. Found it was later changed to 24.",
"StatusCode": "Overridden"
}
{
"ScalableDimension": "ecs:service:DesiredCount",
"Description": "Setting desired count to 18.",
"ResourceId": "service/hub-prod_ecs_cluster/hub-prod_es_blue",
"ActivityId": "e235d7c8-3558-4648-a276-5cb81f1b7af2",
"StartTime": 1519252777.465,
"ServiceNamespace": "ecs",
"EndTime": 1519252843.723,
"Cause": "monitor alarm TargetTracking-service/hub-prod_ecs_cluster/hub-prod_es_blue-AlarmHigh-768f842f-5811-4856-bccf-d73b91f20bfb in state ALARM triggered policy hub-prod_blue_cpu_target_tracking",
"StatusMessage": "Successfully set desired count to 18. Found it was later changed to 21.",
"StatusCode": "Overridden"
}
{
"ScalableDimension": "ecs:service:DesiredCount",
"Description": "Setting desired count to 13.",
"ResourceId": "service/hub-prod_ecs_cluster/hub-prod_es_blue",
"ActivityId": "47835aee-9c64-428d-81b3-0367da773370",
"StartTime": 1519252717.297,
"ServiceNamespace": "ecs",
"EndTime": 1519252784.681,
"Cause": "monitor alarm TargetTracking-service/hub-prod_ecs_cluster/hub-prod_es_blue-AlarmHigh-768f842f-5811-4856-bccf-d73b91f20bfb in state ALARM triggered policy hub-prod_blue_cpu_target_tracking",
"StatusMessage": "Successfully set desired count to 13. Found it was later changed to 18.",
"StatusCode": "Overridden"
}

绿色:

$ aws application-autoscaling describe-scaling-activities --service-namespace ecs \
    --resource-id service/hub-prod_ecs_cluster/hub-prod_es_green \
    | jq '.ScalingActivities | .[] | select(.StartTime >= 1519252380)'
{
"ScalableDimension": "ecs:service:DesiredCount",
"Description": "Setting desired count to 3.",
"ResourceId": "service/hub-prod_ecs_cluster/hub-prod_es_green",
"ActivityId": "4b910f88-87da-4173-899e-246533c4d35c",
"StartTime": 1519253491.575,
"ServiceNamespace": "ecs",
"EndTime": 1519253558.285,
"Cause": "monitor alarm TargetTracking-service/hub-prod_ecs_cluster/hub-prod_es_green-AlarmLow-074e9dde-7936-4b53-ba31-0665190c084f in state ALARM triggered policy hub-prod_green_cpu_target_tracking",
"StatusMessage": "Successfully set desired count to 3. Change successfully fulfilled by ecs.",
"StatusCode": "Successful"
}

0 个答案:

没有答案