AWS ElasticSearch Service-从CF模板设置加密选项

Question

我正在创建一个云形成模板，以在AWS中配置Elasticsearch服务域。

我想将“加密”下的此属性设置为true “到域的所有流量都需要HTTPS”，但是我无法在AWS文档中找到这样做的方法。

用于设置加密属性的其他选项，例如 充分记录了“启用静态数据加密”和“节点到节点加密”。

有人知道如何从CF模板设置“到域的所有流量都需要HTTPS”属性吗？

Answer 1

我这样做的方法是确保安全组仅允许HTTPS（443）访问群集。

不能完全确定这是否是您想要的东西，但是如果不是，请给我更多详细信息，我会看看是否能为您提供帮助。

  mySecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      VpcId: !Ref VpcId
      GroupName: !Ref SecurityGroup
      GroupDescription: !Ref GroupDescription
      SecurityGroupIngress:
        - FromPort: '443'
          IpProtocol: tcp
          ToPort: '443'
          CidrIp: 0.0.0.0/0

Answer 2

以下是我汇总的静态加密以及节点到节点加密的内容：

现在，我不得不添加/编辑大量活动部件，因此我将在此处发布整个模板，以便您可以在不同部分中看到我所做的事情，从而可以提取需要/想要的内容。我使用了条件选项，因为您可以选择启用或不启用它。我还确保在此处和此处添加一些评论。随意剥离它们，但它们根本不会影响模板。

edit ：我知道模板中有一些不需要的部分，但是我想使我的模板看起来很漂亮且井井有条（使用时）。我知道我需要在模板本身内部进行组织，但是就在那里。 ：P

edit2 ：以下模板假定您将使用现有的VPC /子网/安全组，您可以检出我的存储库（在下面的评论中）以获取该模板的其他版本。

我确实验证了此模板，并能够成功构建域。

AWSTemplateFormatVersion: 2010-09-09
Description: >-
  **NOTE** In order to create Elastisearch Domain in AWS using CloudFormation
  verify you have the following Service Role created in IAM!!
  -- AWSServiceRoleForAmazonElasticsearchService --
  If you do not have this Role, create it using the following CLi Command -- 
  aws iam create-service-linked-role --aws-service-name es.amazonaws.com

Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      -
        Label:
          default: "Configure Cluster"
        Parameters:
          - DomainName
          - ElasticsearchVersion
          - ZoneAwareness
          - SnapShotHour
      -
        Label:
          default: "Data Instances"
        Parameters:
          - InstanceType
          - DataInstanceCount
      -
        Label:
          default: "Dedicated Master Instances"
        Parameters:
          - DedicatedMaster
          - MasterInstanceType
          - MasterInstanceCount
      -
        Label:
          default: "Storage Config"
        Parameters:
          - StorageSize
      -
        Label:
          default: "Network Config"
        Parameters:
          - VpcId
          - SubNet1
          - SubNet2
          - SecurityGroup
      -
        Label:
          default: "Encryption Settings"
        Parameters:
          - EncryptionAtRest
          - KmsKey
          - NodetoNode
      -
        Label:
          default: "IAM User Restriction Policy"
        Parameters:
          - IamUserArn
    ParameterLabels:
      DomainName:
        default: "Name of the ElasticSearch Domain (lowecase, no spaces) - If you don't specify a name, AWS CloudFormation generates a unique physical ID and uses that ID for the domain name"
      ElasticsearchVersion:
        default: "Select the ElasticSearch version desired"
      InstanceType:
        default: "Instance Size of Data Instances"
      DataInstanceCount:
        default: "Number of Data Instances Required"
      DedicatedMaster:
        default: "Select if a Dedicated Master Instance is required"
      MasterInstanceType:
        default: "Instance Size of Master Instances"
      MasterInstanceCount:
        default: "How many Dedicated Master Instances are needed? (0, 3 or 5)"
      StorageSize:
        default: "Storage Size in GB"
      VpcId:
        default: "Select the VPC to deploy into (must already exist)"
      SubNet1:
        default: "Select the First Subnet"
      SubNet2:
        default: "Select the Second Subnet"
      SecurityGroup:
        default: "Select the Security Group"
      IamUserArn:
        default: "Enter the ARN for the IAM User to give initial access to the stack"
      ZoneAwareness:
        default: "Enable Zone Awareness (Availability Zone Replication) (recommended)"
      SnapShotHour:
        default: "Set the hour to run the Automated Snapshot (0-23) (Default: UTC Timezone)"
      EncryptionAtRest:
        default: "Enable Encryption at Rest"
      KmsKey:
        default: "If Encryption at Rest is enabled, supply the KMS Key used for encryption"
      NodetoNode:
        default: "Enable Node to Node Encryption"

Parameters:
  DomainName:
    Type: String
    Default: "elasticsearchstack-cf"
    MaxLength: '128' 
    ConstraintDescription: "Must be lowercase, numbers/letters and/or a dash"
  SnapShotHour:
    Type: Number
    Default: 0
    MinValue: 0
    MaxValue: 23
  ElasticsearchVersion:
    Type: String
    Default: 7.1
    AllowedValues: [7.1, 6.8, 6.7, 6.6, 6.5] # Remove this line for free-form number entry
  InstanceType:
    Type: String
    Default: r5.large.elasticsearch
    AllowedValues: [t2.small.elasticsearch, t2.medium.elasticsearch,
      c4.large.elasticsearch, c4.xlarge.elasticsearch, c4.2xlarge.elasticsearch, c4.4xlarge.elasticsearch, c4.8xlarge.elasticsearch,
      c5.large.elasticsearch, c5.xlarge.elasticsearch, c5.2xlarge.elasticsearch, c5.4xlarge.elasticsearch, c5.9xlarge.elasticsearch, c5.18xlarge.elasticsearch,
      m3.medium.elasticsearch, m3.large.elasticsearch, m3.xlarge.elasticsearch, m3.2xlarge.elasticsearch,
      m4.large.elasticsearch, m4.xlarge.elasticsearch, m4.2xlarge.elasticsearch, m4.4xlarge.elasticsearch, m4.10xlarge.elasticsearch,
      m5.large.elasticsearch, m5.xlarge.elasticsearch, m5.2xlarge.elasticsearch, m5.4xlarge.elasticsearch, m5.12xlarge.elasticsearch,
      r3.large.elasticsearch, r3.xlarge.elasticsearch, r3.2xlarge.elasticsearch, r3.4xlarge.elasticsearch, r3.8xlarge.elasticsearch,
      r4.large.elasticsearch, r4.xlarge.elasticsearch, r4.2xlarge.elasticsearch, r4.4xlarge.elasticsearch, r4.16xlarge.elasticsearch,
      r5.large.elasticsearch, r5.xlarge.elasticsearch, r5.2xlarge.elasticsearch, r5.4xlarge.elasticsearch, r5.12xlarge.elasticsearch,
      i2.xlarge.elasticsearch, i2.2xlarge.elasticsearch,
      i3.large.elasticsearch, i3.xlarge.elasticsearch, i3.2xlarge.elasticsearch, i3.4xlarge.elasticsearch, i3.8xlarge.elasticsearch, i3.16xlarge.elasticsearch]
    ConstraintDescription: "Must be a valid EC2 Elasticsearch instance type."
  DataInstanceCount:
    Type: Number
    Default: 2
    AllowedValues: [2, 4, 6, 8, 10] # Remove this line for free-form number entry
  MasterInstanceType:
    Type: String
    Default: r5.large.elasticsearch
    AllowedValues: [t2.small.elasticsearch, t2.medium.elasticsearch,
      c4.large.elasticsearch, c4.xlarge.elasticsearch, c4.2xlarge.elasticsearch, c4.4xlarge.elasticsearch, c4.8xlarge.elasticsearch,
      c5.large.elasticsearch, c5.xlarge.elasticsearch, c5.2xlarge.elasticsearch, c5.4xlarge.elasticsearch, c5.9xlarge.elasticsearch, c5.18xlarge.elasticsearch,
      m3.medium.elasticsearch, m3.large.elasticsearch, m3.xlarge.elasticsearch, m3.2xlarge.elasticsearch,
      m4.large.elasticsearch, m4.xlarge.elasticsearch, m4.2xlarge.elasticsearch, m4.4xlarge.elasticsearch, m4.10xlarge.elasticsearch,
      m5.large.elasticsearch, m5.xlarge.elasticsearch, m5.2xlarge.elasticsearch, m5.4xlarge.elasticsearch, m5.12xlarge.elasticsearch,
      r3.large.elasticsearch, r3.xlarge.elasticsearch, r3.2xlarge.elasticsearch, r3.4xlarge.elasticsearch, r3.8xlarge.elasticsearch,
      r4.large.elasticsearch, r4.xlarge.elasticsearch, r4.2xlarge.elasticsearch, r4.4xlarge.elasticsearch, r4.16xlarge.elasticsearch,
      r5.large.elasticsearch, r5.xlarge.elasticsearch, r5.2xlarge.elasticsearch, r5.4xlarge.elasticsearch, r5.12xlarge.elasticsearch,
      i2.xlarge.elasticsearch, i2.2xlarge.elasticsearch,
      i3.large.elasticsearch, i3.xlarge.elasticsearch, i3.2xlarge.elasticsearch, i3.4xlarge.elasticsearch, i3.8xlarge.elasticsearch, i3.16xlarge.elasticsearch]
    ConstraintDescription: "Must be a valid EC2 Elasticsearch instance type."
  MasterInstanceCount:
    Type: Number
    Default: 0
    AllowedValues: [0, 3, 5] # Remove this line for free-form number entry
  VpcId:
    Type: AWS::EC2::VPC::Id
    ConstraintDescription: "Must be the VPC ID of an existing Virtual Private Cloud."
  SubNet1:
    Type: AWS::EC2::Subnet::Id
    ConstraintDescription: "Must be the Subnet ID of an existing Subnet."
  SubNet2:
    Type: AWS::EC2::Subnet::Id
    ConstraintDescription: "Must be the Subnet ID of an existing Subnet."    
  SecurityGroup:
    Type: AWS::EC2::SecurityGroup::Id
    ConstraintDescription: "Must be and existing Security Group."    
  DedicatedMaster:
    Description: True or False
    Type: String
    Default: False
    AllowedValues:
      - True
      - False
  StorageSize:
    Type: Number
    Default: 20
    MinValue: 10 # Remove this line for free-form number entry (suggested to keep this line)
    MaxValue: 1000 # Remove this line for free-form number entry
  IamUserArn:
    Type: String
    Default: "arn:aws:iam::<AccountNumber>:user/<username>"
  ZoneAwareness:
    Description: True or False
    Type: String
    Default: True
    AllowedValues:
      - True
      - False
  EncryptionAtRest:
    Description: True or False
    Type: String
    Default: False
    AllowedValues:
      - True
      - False
  KmsKey:
    Type: String
  NodetoNode:
    Description: True or False
    Type: String
    Default: False
    AllowedValues:
      - True
      - False
Conditions: # Checks to see if Conditional Values are True
  DedicatedMasterYes: !Equals [ !Ref DedicatedMaster, True]
  EncryptionAtRestYes: !Equals [ !Ref EncryptionAtRest, True]

Resources:
  ElasticsearchDomain:
    Type: AWS::Elasticsearch::Domain
    Properties:
      DomainName: !Ref DomainName
      ElasticsearchVersion: !Ref ElasticsearchVersion
      ElasticsearchClusterConfig: 
        DedicatedMasterEnabled: !Ref DedicatedMaster
        InstanceCount: !Ref DataInstanceCount
        ZoneAwarenessEnabled: !Ref ZoneAwareness
        InstanceType: !Ref InstanceType
        DedicatedMasterType: # If Dedicated Master is True, then use !Ref, if not, use NoValue (NULL) for both settings below
          !If [DedicatedMasterYes, !Ref MasterInstanceType, !Ref "AWS::NoValue"]
        DedicatedMasterCount: 
          !If [DedicatedMasterYes, !Ref MasterInstanceCount, !Ref "AWS::NoValue"]
      EBSOptions:
        EBSEnabled: True
        Iops: 0
        VolumeSize: !Ref StorageSize
        VolumeType: "gp2"
      SnapshotOptions:
        AutomatedSnapshotStartHour: !Ref SnapShotHour
      AccessPolicies:
        Version: 2012-10-17
        Statement:
          - Effect: Deny
            Principal:
              AWS: !Ref IamUserArn
            Action: 'es:*'
            Resource: "*"
      AdvancedOptions: # BOTH of these settingsd are REQUIRED (regardless of what the documentation states) - Bug filed: https://forums.aws.amazon.com/thread.jspa?messageID=768527
        rest.action.multi.allow_explicit_index: 'true'
        indices.fielddata.cache.size: !Ref "AWS::NoValue"
      VPCOptions:
        SubnetIds:
          - !Ref SubNet1
          - !Ref SubNet2
        SecurityGroupIds:
          - !Ref SecurityGroup
      EncryptionAtRestOptions: # If Encryption At Rest is True, then use !Ref, if not, use NoValue (NULL) for both settings below
          !If [EncryptionAtRestYes, !Ref EncryptionAtRest, !Ref "AWS::NoValue"]
      KmsKeyId:
          !If [EncryptionAtRestYes, !Ref KmsKey, !Ref "AWS::NoValue"]
      NodeToNodeEncryptionOptions:
        Enabled: !Ref NodetoNode

Outputs:
  DomainArn:
    Value: !GetAtt ElasticsearchDomain.DomainArn
  DomainEndpoint:
    Value: !GetAtt ElasticsearchDomain.DomainEndpoint
  SecurityGroupId:
    Value: !Ref SecurityGroup
  SubnetId1:
    Value: !Ref SubNet1
  SubnetId2:
    Value: !Ref SubNet2

Answer 3

由于该功能是最近发布的，它尚未在CloudFormation或Terraform中可用，但是我们可以通过在CF模板中使用自定义资源来实现相同功能。该自定义资源不过是一个lambda函数，它将作为部分云形成堆栈的创建而执行。

完全自动化可以分为3个部分，

1. Lambda函数开发
2. Lambda函数创建
3. 将Lambda函数配置作为Cloudformation堆栈中的自定义资源

Lambda函数开发

如here所述，请按照以下步骤创建lambda函数。

创建一个空文件夹，将其用于放置Lambda源。然后使用pip将crhelper安装到该文件夹​​中，并创建lambda_function.py文件以将您的资源代码放入其中。

mkdir sum_function
cd sum_function
pip install -t . crhelper
# on some systems pip may fail with a distutils error, if you run into this, try 
running pip with the –system argument
# pip install —system -t . crhelper
touch lambda_function.py

现在打开lambda_function.py文件，并将以下代码放入lambda_function.py文件中。

from crhelper import CfnResource
import boto3
import logging

logger = logging.getLogger(__name__)
helper = CfnResource()

@helper.create
@helper.update
def update_es_domain_config(event, context):
    logger.info("Updating elasticsearch domain config with HTTPS")
    es = boto3.client('es')
    response = es.update_elasticsearch_domain_config(
        DomainName=event['ResourceProperties']['ESDomainName'],
        DomainEndpointOptions={
            'EnforceHTTPS': True,
            'TLSSecurityPolicy': 'Policy-Min-TLS-1-0-2019-07'
        }
    )
    logger.info("results:", response)
@helper.delete
def no_op(_, __):
    pass

def handler(event, context):
    helper(event, context)

之后，压缩整个目录

zip -r ../sum.zip ./

Lambda函数创建

现在创建lambda函数，我同样使用CLI。根据您的环境替换这些值。

aws lambda create-function \
    --function-name "update-es-domain" \
    --handler "lambda_function.handler" \
    --timeout 900 \
    --zip-file fileb://../sum.zip \
    --runtime python3.7 \
    --role "arn:aws:iam::123412341234:role/lambda-cli-role"

您需要从输出中删除FunctionArn以便在下一步中使用。

Lambda函数配置为Cloudformation堆栈中的自定义资源

现在在CF模板的“资源”部分中，在Elasticsearch资源之后，添加自定义资源，如下所示。将服务令牌替换为您在上一步中获得的functionArn作为输出。将ESmainName替换为要修改的elasticsearch域。

  ESDomainUpdate:
    Type: "Custom::elasticsearch"
    Properties:
      ServiceToken: "arn:aws:lambda:us-east-1:123412341234:function:update-es-domain"
      ESDomainName: "advantagehelp"

PFB完整的Cloudformation模板以供参考。

AWSTemplateFormatVersion: 2010-09-09
Description: AWS CloudFormation templates to create AWS Elasticsearch Service domain.
Parameters:
  NodeType:
    Description: The node type to be provisioned for the Elasticsearch cluster
    Type: String
    Default: m4.large.elasticsearch
    AllowedValues:
      - m4.large.elasticsearch
      - m4.xlarge.elasticsearch
      - c4.large.elasticsearch
      - c4.xlarge.elasticsearch
      - r4.large.elasticsearch
      - r4.xlarge.elasticsearch
    ConstraintDescription: must be a valid Elasticsearch node type.
  NodeCount:
    Description: The number of nodes in the Elasticsearch cluster.
    Type: Number
    Default: '1'
  AllowedESGetIpList:
    Type: CommaDelimitedList
    Description: List of ip and cidr for elasticsearch retrieval
  AllowedESUpdateIpList:
    Type: CommaDelimitedList
    Description: List of ip and cidr for elasticsearch update
  ESDomainName:
    Type: String
    Description: Name of Elastic Search Domain
    Default: advantagehelp
Resources:
  ElasticsearchDomain:
    Type: 'AWS::Elasticsearch::Domain'
    Metadata:
      cfn_nag:
        rules_to_suppress:
          - id: W28
            reason: "Needed to name elasticsearch domain"
    Properties:
      DomainName: !Ref ESDomainName
      ElasticsearchClusterConfig:
        DedicatedMasterEnabled: false
        InstanceCount: !Ref NodeCount
        ZoneAwarenessEnabled: false
        InstanceType: !Ref NodeType
      ElasticsearchVersion: '6.4'
      AccessPolicies:
        Version: 2012-10-17
        Statement:
          - Action:
              - 'es:ESHttpGet'
              - 'es:ESHttpHead'
            Principal: '*'
            Effect: Allow
            Resource: !Sub arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/${ESDomainName}/*
            Condition:
              IpAddress:
                'aws:SourceIp': !Ref AllowedESGetIpList
          - Action:
              - 'es:ESHttpGet'
              - 'es:ESHttpHead'
              - 'es:ESHttpPost'
              - 'es:ESHttpPut'
            Principal: '*'
            Effect: Allow
            Resource: !Sub arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/${ESDomainName}/*
            Condition:
              IpAddress:
                'aws:SourceIp': !Ref AllowedESUpdateIpList
      EBSOptions:
        EBSEnabled: true
        Iops: 0
        VolumeSize: 10
        VolumeType: gp2
      SnapshotOptions:
        AutomatedSnapshotStartHour: 1
      AdvancedOptions:
        rest.action.multi.allow_explicit_index: "true"
  ESDomainUpdate:
    Type: "Custom::elasticsearch"
    Properties:
      ServiceToken: "arn:aws:lambda:us-east-1:123412341234:function:update-es-domain"
      ESDomainName: "advantagehelp"
    DependsOn: ElasticsearchDomain
Outputs:
  KibanaURL:
    Description: Kibana URL
    Value: !Join
      - ''
      - - !GetAtt
          - ElasticsearchDomain
          - DomainEndpoint
        - /_plugin/kibana/
  ElasticsearchEndpoint:
    Description: Elasticsearch domain endpoint
    Value: !GetAtt
      - ElasticsearchDomain
      - DomainEndpoint
  ElasticsearchDomainARN:
    Description: Elasticsearch domain ARN
    Value: !GetAtt
      - ElasticsearchDomain
      - DomainArn