Question

我正在尝试为我的EMR集群创建SNS警报，因此当EMR集群发生故障时，我应该得到通知。

但是我的问题是我无法在CloudWatch警报中将群集ID作为JobFlowId传递。

我正在使用CloudFomartion Templet创建所有资源。

当我使用REF引用群集ID时，出现以下错误。

Error
Template validation error: Template format error: Unresolved resource dependencies [FinancialLineItemEmrCluster] in the Resources block of the template

这是我的庙宇。

在创建资源JobFlowId

时，我在EMRAlarm有一个特别的问题。

AWSTemplateFormatVersion: '2010-09-09'
Description: TRF SDI Optmization Full File  Creation

Parameters:
  AppName:
    Default: trfsdioptmization
    Description: trfsdioptmization.
    Type: String
  Environment:
    Type: String
    Default: nonprod
  FinancialIdentifier:
    Type: String
    Default: 123456789
  ApplicationAssetInsightId:
    Type: String
    Default: 12345678
  EnvironmentType:
    Type: String
    AllowedValues:
      - "prod"
      - "PRE-PRODUCTION"
      - "QUALITY ASSURANCE"
      - "nonprod"
    Default: nonprod
  ResourceOwner:
    Type: String
    Default: sudarshan.kumar@abcd.com
  EnvironmentPhase:
    Type: String
    Default: nonprod
  RegionAbbreviation:
    Default: us-east-1
    Description: Region Abbreviation e.g. us-east-1 for us-east
    Type: String

Resources:
 TRFSDIFullfileGeneration: 
  Type: "AWS::DataPipeline::Pipeline"
  #DeletionPolicy: Retain
  Properties: 
    Name: !Sub "${ApplicationAssetInsightId}-tr-fr-${EnvironmentPhase}-${RegionAbbreviation}-${AppName}-DataPipeline"
    Description: "Pipeline to create full file for TRFSDI full file Optmization"
    Activate: false
    PipelineObjects: 
      - 
        Id: "FinancialLineItemActivity"
        Name: "FinancialLineItemActivity"
        Fields: 
          - 
            Key: "type"
            StringValue: "EmrActivity"
          - 
            Key: "runsOn"
            RefValue: "FinancialLineItemEmrCluster"
          - 
            Key: "step"
            StringValue: "command-runner.jar,spark-submit,--master,yarn-cluster,--deploy-mode,cluster,--class,start.EntryFileCreation,s3://205147-trf-fr-nonprdo-us-east-1-trfsdioptmization/AJAR/SparkJob-0.1-jar-with-dependencies.jar,FinancialLineItem"

        Id: "Default"
        Name: "Default"
        Fields: 
          - 
            Key: "type"
            StringValue: "Default"
          - 
            Key: "scheduleType"
            StringValue: "ONDEMAND"
          - 
            Key: "failureAndRerunMode"
            StringValue: "CASCADE"
          - 
            Key: "role"
            StringValue: "DataPipelineDefaultRole"
          - 
            Key: "resourceRole"
            StringValue: "DataPipelineDefaultResourceRole"
          - 
            Key: "pipelineLogUri"
            StringValue: "s3://205147-tr-fr-nonprod-us-east-1-trfsdioptmization/EMRLOGS"
      - 
        Id: "FinancialLineItemEmrCluster"
        Name: "FinancialLineItemEmrCluster"
        Fields: 
          - 
            Key: "terminateAfter"
            StringValue: "30 Minutes"
          - 
            Key: "releaseLabel"
            StringValue: "emr-5.9.0"
          - 
            Key: "masterInstanceType"
            StringValue: "m3.xlarge"
          - 
            Key: "coreInstanceType"
            StringValue: "m3.2xlarge"
          - 
            Key: "coreInstanceCount"
            StringValue: "2"
          - 
            Key: "type"
            StringValue: "EmrCluster"
          - 
            Key: "applications"
            StringValue: "spark"
          - 
            Key: "subnetId"
            StringValue: "subnet-86febcab"
          - 
            Key: "onSuccess"
            RefValue: "FinancialLineItem_Success"
          - 
            Key: "onFail"
            RefValue: "FinancialLineItem_Fail"
 EMRAlarm:
   Type: AWS::CloudWatch::Alarm
   Properties:
    AlarmDescription: "Raise alarm if apps running on EMR cluster is killed"
    Namespace: AWS/ElasticMapReduce
    MetricName: AppsKilled
    Dimensions:
    - Name: 205147-TRFSDIOPTmization
      JobFlowId: !Ref FinancialLineItemEmrCluster
    Statistic: Average
    Period: 15
    EvaluationPeriods: '1'
    ComparisonOperator: GreaterThanOrEqualToThreshold
    Threshold: 1
    AlarmActions:
    - "AWSTRF_HEALTH"

Answer 1

模板中没有FinancialLineItemEmrCluster类型的资源。您在CloudWatch警报中引用的是FinancialLineItemEmrCluster。从上下文来看，我假设您正在尝试引用EMR作业。但是，由于模板j-1ABCD123AB1A中没有参数或资源，因此无法访问is。

EMR群集ID通常看起来像这样：FinancialLineItemEmrCluster: Description: 'Your EMR cluster id. Example: j-1ABCD123AB1A' Type: String。您有几种选择：

如果该群集位于另一个模板中，则可以在该堆栈中创建一个Blake2，然后在模板中使用CloudFormation export进行导入。

或者，您可以在模板中使用Parameter并以这种方式传递ClusterId。将此添加为参数：

示例：

FinancialLineItemEmrCluster: 
  Type: AWS::EMR::Cluster
  Properties: 
    Instances: 
      MasterInstanceGroup: 
        InstanceCount: 1
        InstanceType: "m3.xlarge"
        Market: "ON_DEMAND"
        Name: "Master"
      CoreInstanceGroup: 
        InstanceCount: 2
        InstanceType: "m3.xlarge"
        Market: "ON_DEMAND"
        Name: "Core"
      TerminationProtected: true
    Name: "TestCluster"
    JobFlowRole: "EMR_EC2_DefaultRole"
    ServiceRole: "EMR_DefaultRole"
    ReleaseLabel: "emr-4.2.0"

第三种选择是将其硬编码到模板中。

无论如何，您不能直接引用另一个堆栈中的资源。

如果您根本没有创建EMR :: Cluster，并且没有群集ID，则需要首先创建一个。您可以使用!ImportValue资源向模板中添加一个：

{{1}}

如何在CloudWatch警报中传递EMR群集ID

1 个答案: