从负载均衡的EC2 Web服务器向排队的EC2工作人员发送消息

时间：2017-03-10 21:26:10

标签： amazon-web-services amazon-ec2 amazon-sqs amazon-cloudformation amazon-elb

我有一个相当简单的CloudFormation模板，我一直在努力，最初只包含一组负载均衡的EC2 Web服务器。但是，现在我希望这些Web服务器能够将消息发送到SQS消息队列，然后将工作交给某些EC2工作服务器。

这是我的设计：

enter image description here

这是我当前的JSON模板：

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "Stack for MySite",
    "Parameters": {
        "KeyName": {
            "Description": "Key Pair name",
            "Type": "AWS::EC2::KeyPair::KeyName",
            "Default": "mykey"
        },
        "SiteID": {
            "Description": "A unique identifier for the site.",
            "Type": "String",
            "AllowedPattern": "[A-Za-z0-9\\-]+",
            "ConstraintDescription": "Only letters, digits or dash allowed."
        },
        "SiteTitle": {
            "Description": "The title of the site.",
            "Type": "String",
            "Default": "MySite"
        },
        "AdminUsername": {
            "Description": "A username for admin.",
            "Type": "String",
            "Default": "admin"
        },
        "AdminPassword": {
            "Description": "A password for admin.",
            "Type": "String",
            "NoEcho": "true"
        },
        "AdminEMail": {
            "Description": "The email address of the administrator.",
            "Type": "String"
        }
    },
    "Mappings": {
        "EC2RegionMap": {
            "ap-northeast-1": {"AmazonLinuxAMIHVMEBSBacked64bit": "ami-cbf90ecb"},
            "ap-southeast-1": {"AmazonLinuxAMIHVMEBSBacked64bit": "ami-68d8e93a"},
            "ap-southeast-2": {"AmazonLinuxAMIHVMEBSBacked64bit": "ami-fd9cecc7"},
            "eu-central-1": {"AmazonLinuxAMIHVMEBSBacked64bit": "ami-a8221fb5"},
            "eu-west-1": {"AmazonLinuxAMIHVMEBSBacked64bit": "ami-a10897d6"},
            "sa-east-1": {"AmazonLinuxAMIHVMEBSBacked64bit": "ami-b52890a8"},
            "us-east-1": {"AmazonLinuxAMIHVMEBSBacked64bit": "ami-1ecae776"},
            "us-west-1": {"AmazonLinuxAMIHVMEBSBacked64bit": "ami-d114f295"},
            "us-west-2": {"AmazonLinuxAMIHVMEBSBacked64bit": "ami-e7527ed7"}
        }
    },
    "Resources": {
        "VPC": {
            "Type": "AWS::EC2::VPC",
            "Properties": {
                "CidrBlock": "172.31.0.0/16",
                "EnableDnsHostnames": "true"
            }
        },
        "InternetGateway": {
            "Type": "AWS::EC2::InternetGateway",
            "Properties": {
            }
        },
        "VPCGatewayAttachment": {
            "Type": "AWS::EC2::VPCGatewayAttachment",
            "Properties": {
                "VpcId": {"Ref": "VPC"},
                "InternetGatewayId": {"Ref": "InternetGateway"}
            }
        },
        "SubnetA": {
            "Type": "AWS::EC2::Subnet",
            "Properties": {
                "AvailabilityZone": {"Fn::Select": ["0", {"Fn::GetAZs": ""}]},
                "CidrBlock": "172.31.38.0/24",
                "VpcId": {"Ref": "VPC"}
            }
        },
        "SubnetB": {
            "Type": "AWS::EC2::Subnet",
            "Properties": {
                "AvailabilityZone": {"Fn::Select": ["1", {"Fn::GetAZs": ""}]},
                "CidrBlock": "172.31.37.0/24",
                "VpcId": {"Ref": "VPC"}
            }
        },
        "WebServerRouteTable": {
            "Type": "AWS::EC2::RouteTable",
            "Properties": {
                "VpcId": {"Ref": "VPC"}
            }
        },
        "RouteTableAssociationA": {
            "Type": "AWS::EC2::SubnetRouteTableAssociation",
            "Properties": {
                "SubnetId": {"Ref": "SubnetA"},
                "RouteTableId": {"Ref": "WebServerRouteTable"}
            }
        },
        "RouteTableAssociationB": {
            "Type": "AWS::EC2::SubnetRouteTableAssociation",
            "Properties": {
                "SubnetId": {"Ref": "SubnetB"},
                "RouteTableId": {"Ref": "WebServerRouteTable"}
            }
        },
        "RoutePublicNATToInternet": {
            "Type": "AWS::EC2::Route",
            "Properties": {
                "RouteTableId": {"Ref": "WebServerRouteTable"},
                "DestinationCidrBlock": "0.0.0.0/0",
                "GatewayId": {"Ref": "InternetGateway"}
            },
            "DependsOn": "VPCGatewayAttachment"
        },
        "NetworkAcl": {
            "Type": "AWS::EC2::NetworkAcl",
            "Properties": {
                "VpcId": {"Ref": "VPC"}
            }
        },
        "SubnetNetworkAclAssociationA": {
            "Type": "AWS::EC2::SubnetNetworkAclAssociation",
            "Properties": {
                "SubnetId": {"Ref": "SubnetA"},
                "NetworkAclId": {"Ref": "NetworkAcl"}
            }
        },
        "SubnetNetworkAclAssociationB": {
            "Type": "AWS::EC2::SubnetNetworkAclAssociation",
            "Properties": {
                "SubnetId": {"Ref": "SubnetB"},
                "NetworkAclId": {"Ref": "NetworkAcl"}
            }
        },
        "NetworkAclEntryIngress": {
            "Type": "AWS::EC2::NetworkAclEntry",
            "Properties": {
                "NetworkAclId": {"Ref": "NetworkAcl"},
                "RuleNumber": "100",
                "Protocol": "-1",
                "RuleAction": "allow",
                "Egress": "false",
                "CidrBlock": "0.0.0.0/0"
            }
        },
        "NetworkAclEntryEgress": {
            "Type": "AWS::EC2::NetworkAclEntry",
            "Properties": {
                "NetworkAclId": {"Ref": "NetworkAcl"},
                "RuleNumber": "100",
                "Protocol": "-1",
                "RuleAction": "allow",
                "Egress": "true",
                "CidrBlock": "0.0.0.0/0"
            }
        },
        "LoadBalancer": {
            "Type": "AWS::ElasticLoadBalancing::LoadBalancer",
            "Properties": {
                "Subnets": [{"Ref": "SubnetA"}, {"Ref": "SubnetB"}],
                "LoadBalancerName": {"Ref": "SiteID"},
                "Listeners": [{
                    "InstancePort": "80",
                    "InstanceProtocol": "HTTP",
                    "LoadBalancerPort": "80",
                    "Protocol": "HTTP"
                }],
                "HealthCheck": {
                    "HealthyThreshold": "2",
                    "Interval": "5",
                    "Target": "TCP:80",
                    "Timeout": "3",
                    "UnhealthyThreshold": "2"
                },
                "SecurityGroups": [{"Ref": "LoadBalancerSecurityGroup"}],
                "Scheme": "internet-facing",
                "CrossZone": "true"
            },
            "DependsOn": "VPCGatewayAttachment"
        },
        "LoadBalancerSecurityGroup": {
            "Type": "AWS::EC2::SecurityGroup",
            "Properties": {
                "GroupDescription": "mysite-elb-sg",
                "VpcId": {"Ref": "VPC"},
                "SecurityGroupIngress": [{
                    "CidrIp": "0.0.0.0/0",
                    "FromPort": 80,
                    "IpProtocol": "tcp",
                    "ToPort": 80
                }]
            }
        },
        "WebServerSecurityGroup": {
            "Type": "AWS::EC2::SecurityGroup",
            "Properties": {
                "GroupDescription": "mysite-web-server-sg",
                "VpcId": {"Ref": "VPC"},
                "SecurityGroupIngress": [{
                    "CidrIp": "0.0.0.0/0",
                    "FromPort": 22,
                    "IpProtocol": "tcp",
                    "ToPort": 22
                }, {
                    "FromPort": 80,
                    "IpProtocol": "tcp",
                    "SourceSecurityGroupId": {"Ref": "LoadBalancerSecurityGroup"},
                    "ToPort": 80
                }]
            }
        },
        "DatabaseSecurityGroup": {
            "Type": "AWS::EC2::SecurityGroup",
            "Properties": {
                "GroupDescription": "mysite-db-sg",
                "VpcId": {"Ref": "VPC"},
                "SecurityGroupIngress": [{
                    "IpProtocol": "tcp",
                    "FromPort": "3306",
                    "ToPort": "3306",
                    "SourceSecurityGroupId": {"Ref": "WebServerSecurityGroup"}
                }]
            }
        },
        "Database": {
            "Type": "AWS::RDS::DBInstance",
            "Properties": {
                "AllocatedStorage": "5",
                "BackupRetentionPeriod": "0",
                "DBInstanceClass": "db.t2.micro",
                "DBInstanceIdentifier": {"Ref": "SiteID"},
                "DBName": "quantsketch",
                "Engine": "MySQL",
                "MasterUsername": "mysite",
                "MasterUserPassword": "mysite",
                "VPCSecurityGroups": [{"Fn::GetAtt": ["DatabaseSecurityGroup", "GroupId"]}],
                "DBSubnetGroupName": {"Ref": "DBSubnetGroup"}
            },
            "DependsOn": "VPCGatewayAttachment"
        },
        "DBSubnetGroup" : {
            "Type" : "AWS::RDS::DBSubnetGroup",
            "Properties" : {
                "DBSubnetGroupDescription" : "DB subnet group",
                "SubnetIds": [{"Ref": "SubnetA"}, {"Ref": "SubnetB"}]
            }
        },
        "S3Bucket": {
            "Type": "AWS::S3::Bucket",
            "Properties": {
                "BucketName": {"Ref": "SiteID"},
                "WebsiteConfiguration": {
                    "IndexDocument": "index.html"
                }
            }
        },
        "WebServerLaunchConfiguration": {
            "Type": "AWS::AutoScaling::LaunchConfiguration",
            "Metadata": {
                "AWS::CloudFormation::Init": {
                    "config": {
                        "packages": {
                            "yum": {
                                "php": [],
                                "php-mysql": [],
                                "mysql": [],
                                "httpd": []
                            }
                        },
                        "sources": {
                            "/var/www/html": "https://wordpress.org/wordpress-4.2.4.tar.gz"
                        },
                        "files": {
                            "/tmp/config": {
                                "content": {"Fn::Join": ["", [
                                    "#!/bin/bash -ex\n",
                                ]]},
                                "mode": "000500",
                                "owner": "root",
                                "group": "root"
                            }
                        },
                        "commands": {
                            "01_config": {
                                "command": "/tmp/config",
                                "cwd": "/var/www/html/wordpress"
                            }
                        },
                        "services": {
                            "sysvinit": {
                                "httpd": {
                                    "enabled": "true",
                                    "ensureRunning": "true"
                                }
                            }
                        }
                    }
                }
            },
            "Properties": {
                "ImageId": {"Fn::FindInMap": ["EC2RegionMap", {"Ref": "AWS::Region"}, "AmazonLinuxAMIHVMEBSBacked64bit"]},
                "InstanceType": "t2.micro",
                "SecurityGroups": [{"Ref": "WebServerSecurityGroup"}],
                "KeyName": {"Ref": "KeyName"},
                "AssociatePublicIpAddress": true,
                "UserData": {"Fn::Base64": {"Fn::Join": ["", [
                    "#!/bin/bash -ex\n",
                    "yum update -y aws-cfn-bootstrap\n",
                    "/opt/aws/bin/cfn-init -v --stack ", {"Ref": "AWS::StackName"}, " --resource WebServerLaunchConfiguration --region ", {"Ref": "AWS::Region"}, "\n",
                    "/opt/aws/bin/cfn-signal -e $? --stack ", {"Ref": "AWS::StackName"}, " --resource WebServerAutoScalingGroup --region ", {"Ref": "AWS::Region"}, "\n"
                ]]}}
            }
        },
        "WebServerAutoScalingGroup": {
            "Type": "AWS::AutoScaling::AutoScalingGroup",
            "Properties": {
                "LoadBalancerNames": [{"Ref": "LoadBalancer"}],
                "LaunchConfigurationName": {"Ref": "WebServerLaunchConfiguration"},
                "MinSize": "2",
                "MaxSize": "4",
                "DesiredCapacity": "2",
                "Cooldown": "60",
                "HealthCheckGracePeriod": "120",
                "HealthCheckType": "ELB",
                "VPCZoneIdentifier": [{"Ref": "SubnetA"}, {"Ref": "SubnetB"}],
                "Tags": [{
                    "PropagateAtLaunch": true,
                    "Value": "quantsketch",
                    "Key": "Name"
                }]
            },
            "CreationPolicy": {
                "ResourceSignal": {
                    "Timeout": "PT10M"
                }
            },
            "DependsOn": "VPCGatewayAttachment"
        },
        "WebServerScalingUpPolicy": {
            "Type": "AWS::AutoScaling::ScalingPolicy",
            "Properties": {
                "AdjustmentType": "ChangeInCapacity",
                "AutoScalingGroupName": {"Ref": "WebServerAutoScalingGroup"},
                "Cooldown": "60",
                "ScalingAdjustment": "1"
            }
        },
        "WebServerCPUHighAlarm": {
            "Type": "AWS::CloudWatch::Alarm",
            "Properties": {
                "EvaluationPeriods": "1",
                "Statistic": "Average",
                "Threshold": "80",
                "AlarmDescription": "Alarm if CPU load is high.",
                "Period": "60",
                "AlarmActions": [{"Ref": "WebServerScalingUpPolicy"}],
                "Namespace": "AWS/EC2",
                "Dimensions": [{
                    "Name": "AutoScalingGroupName",
                    "Value": {"Ref": "WebServerAutoScalingGroup"}
                }],
                "ComparisonOperator": "GreaterThanThreshold",
                "MetricName": "CPUUtilization"
            }
        },
        "WebServerScalingDownPolicy": {
            "Type": "AWS::AutoScaling::ScalingPolicy",
            "Properties": {
                "AdjustmentType": "ChangeInCapacity",
                "AutoScalingGroupName": {"Ref": "WebServerAutoScalingGroup"},
                "Cooldown": "60",
                "ScalingAdjustment": "-1"
            }
        },
        "WebServerCPULowAlarm": {
            "Type": "AWS::CloudWatch::Alarm",
            "Properties": {
                "EvaluationPeriods": "1",
                "Statistic": "Average",
                "Threshold": "25",
                "AlarmDescription": "Alarm if CPU load is low.",
                "Period": "60",
                "AlarmActions": [{"Ref": "WebServerScalingDownPolicy"}],
                "Namespace": "AWS/EC2",
                "Dimensions": [{
                    "Name": "AutoScalingGroupName",
                    "Value": {"Ref": "WebServerAutoScalingGroup"}
                }],
                "ComparisonOperator": "LessThanThreshold",
                "MetricName": "CPUUtilization"
            }
        }
    },
    "Outputs": {
        "URL": {
            "Value": {"Fn::Join": ["", ["http://", {"Fn::GetAtt": ["LoadBalancer", "DNSName"]}, "/quantsketch"]]},
            "Description": "QuantSketch URL"
        }
    }
}

不幸的是，我无法找到任何解决此堆栈架构的示例模板，即使我原本期望它是一个常见问题。

如何连接这两个群集，以便Web服务器可以将工作卸载到工作服务器上？任何包含示例，教程或其他任何内容的建议都会有很长的路要走。

1 个答案:

答案 0 :(得分：1)

通常，您可以通过调用负载平衡计算机和工作计算机上运行的应用程序中的AWS API / SDK来发送和接收来自SQS的消息来实现此目的。

选项A：通过SQS直接发送/接收

选项B：通过SNS发布者发布到SQS订阅者的发布/订阅

另一个选择是首先通过SNS发布您的消息，然后通过SQS订阅它。如果您有一个用例，这可以为您提供额外的好处，即能够为邮件订阅不同的应用程序（甚至是人，通过短信/电子邮件）。

  • Web服务器准备您的消息并致电SNS Publish以将您的消息发布给给定SNS主题的所有订阅者。
  • SNS将消息发布给给定SNS主题的所有订阅者;您的队列就是这样一个订阅者，因此消息最终会出现在您的队列中。
  • 工作人员调用SQS ReceiveMessage从队列中接收最多10封邮件，并对其进行处理。
  • 工作人员在处理后调用SQS DeleteMessage，以便不重试该消息。如果您想减少对SQS的调用，也可以致电SQS DeleteMessageBatch。

讨论

您的Web服务器和工作组堆栈都需要能够与您的VPC通信才能到达SNS和/或SQS服务。您可以通过使用NAT Gateways来实现此目的，以允许您的实例与全球互联网进行通信。

您需要提前配置SNS / SQS资源。您可以通过CloudFormation模板执行此操作，也可以通过SNS / SQS API执行此操作。

您没有提到您的应用程序使用的语言;您将需要阅读您将使用的特定SDK的文档，该文档与我在此答案中链接的API相匹配。

要授予您的应用程序访问您的AWS账户中这些资源的权限，我建议您创建一个IAM instance profile，该权限至少具有读取和写入此SQS队列的权限，如果您选择，还可以选择该SNS主题与酒吧/子路线。如果您未明确向应用程序提供凭据，则凭据链将转至IAM实例配置文件并继承。您可以将这些IAM资源创建为CloudFormation模板的一部分。

还有几个最终的哲学笔记：

  • 不要将此问题视为连接群集;在这两种情况下，他们只会了解SNS / SQS。这里的目标是您的网络服务器不了解您的员工，反之亦然。

  • 我认为你过于宽泛地考虑这个问题，这就是你找不到与之相关的资源的原因。将您的查询分解为较小的工作单元，例如“如何向SQS发送消息？”或“我如何从SQS收到消息？”我想你会有更好的运气。

进一步阅读

相关问题
最新问题