Question

是否可以通过cloudformation模板创建Athena视图。我可以使用Athena仪表板创建视图，但是我想使用CF模板以编程方式执行此操作。在AWS文档中找不到任何详细信息，因此不确定是否受支持。

谢谢。

Answer 1

可以使用CloudFormation创建视图，这非常非常复杂。雅典娜视图存储在Glue数据目录中，就像数据库和表一样。实际上，Athena视图是Glue Data Catalog中的表，只是内容略有不同。

请参见此答案以获取有关如何以编程方式创建视图的完整描述，并且您会了解其复杂性：Create AWS Athena view programmatically –可以将其映射到CloudFormation，但我不建议这样做。

如果要使用CloudFormation创建数据库和表，则资源为AWS::Glue::Database和AWS::Glue::Table。

Answer 2

我认为目前从 Athena 模板创建 CloudFormation 视图的最佳方法是使用自定义资源和 Lambda。我们必须提供视图创建和删除的方法。例如，可以定义使用 crhelper 库 Lambda：

from __future__ import print_function
from crhelper import CfnResource
import logging
import os
import boto3

logger = logging.getLogger(__name__)
helper = CfnResource(json_logging=False, log_level='DEBUG', boto_level='CRITICAL', sleep_on_delete=120)

try:
    client = boto3.client('athena')
    ATHENA_WORKGROUP = os.environ['athena_workgroup']
    DATABASE = os.environ['database']
    QUERY_CREATE = os.environ['query_create']
    QUERY_DROP = os.environ['query_drop']
except Exception as e:
    helper.init_failure(e)

@helper.create
@helper.update
def create(event, context):
    logger.info("View creation started")

    try:
        executionResponse = client.start_query_execution(
            QueryString=QUERY_CREATE,
            QueryExecutionContext={'Database': DATABASE},
            WorkGroup='AudienceAthenaWorkgroup'
        )
        logger.info(executionResponse)

        response = client.get_query_execution(QueryExecutionId=executionResponse['QueryExecutionId'])
        logger.info(response)

        if response['QueryExecution']['Status']['State'] == 'FAILED':
            logger.error("Query failed")
            raise ValueError("Query failed")

        helper.Data['success'] = True
        helper.Data['id'] = executionResponse['QueryExecutionId']
        helper.Data['message'] = 'query is running'

    except Exception as e:
        print(f"An exception occurred: {e}")

    if not helper.Data.get("success"):
        raise ValueError("Creating custom resource failed.")

    return


@helper.delete
def delete(event, context):
    logger.info("View deletion started")

    try:
        executionResponse = client.start_query_execution(
            QueryString=QUERY_DROP,
            QueryExecutionContext={'Database': DATABASE},
            WorkGroup='AudienceAthenaWorkgroup'
        )
        logger.info(executionResponse)

    except Exception as e:
        print("An exception occurred")
        print(e)

@helper.poll_create
def poll_create(event, context):
    logger.info("Pol creation")

    response = client.get_query_execution(QueryExecutionId=event['CrHelperData']['id'])

    logger.info(f"Poll response: {response}")

    # There are 3 types of state of query
    # if state is failed - we stop and fail creation
    # if state is queued - we continue polling in 2 minutes
    # if state is succeeded - we stop and succeed creation
    if 'FAILED' == response['QueryExecution']['Status']['State']:
        logger.error("Query failed")
        raise ValueError("Query failed")

    if 'SUCCEEDED' == response['QueryExecution']['Status']['State']:
        logger.error("Query SUCCEEDED")
        return True

    if 'QUEUED' == response['QueryExecution']['Status']['State']:
        logger.error("Query QUEUED")
        return False

    # Return a resource id or True to indicate that creation is complete. if True is returned an id
    # will be generated
    # Return false to indicate that creation is not complete and we need to poll again
    return False

def handler(event, context):
    helper(event, context)

用于视图创建/更新/删除的 Athena 查询作为环境参数传递给 Lambda。在 CloudFormation 模板中，我们必须定义调用提到的 Python 代码并创建/更新/删除 Athena 视图的 Lambda。例如

  AthenaCommonViewLambda:
    Type: 'AWS::Lambda::Function'
    DependsOn: [CreateAthenaViewLayer, CreateAthenaViewLambdaRole]
    Properties:
      Environment:
        Variables:
          athena_workgroup: !Ref AudienceAthenaWorkgroup
          database:
            Ref: DatabaseName
          query_create: !Sub >-
            CREATE OR REPLACE VIEW ${TableName}_view AS
            SELECT field1, field2, ...
            FROM ${DatabaseName}.${TableName}
          query_drop: !Sub DROP VIEW IF EXISTS ${TableName}_common_view
      Code:
        S3Bucket: !Ref SourceS3Bucket
        S3Key: createview.zip
      FunctionName: !Sub '${AWS::StackName}_create_common_view'
      Handler: createview.handler
      MemorySize: 128
      Role: !GetAtt CreateAthenaViewLambdaRole.Arn
      Runtime: python3.8
      Timeout: 60
      Layers:
        - !Ref CreateAthenaViewLayer

  AthenaCommonView:
    Type: 'Custom::AthenaCommonView'
    Properties:
      ServiceToken: !GetAtt AthenaCommonViewLambda.Arn

Answer 3

通常，CloudFormation用于以可重复的方式部署基础架构。这对数据库中的数据不太适用，该数据通常独立存在于其他基础结构中。

对于Amazon Athena，AWS CloudFormation仅支持：

数据目录
命名查询
工作组

最接近您要求的是命名查询，（我认为）该查询可以存储可以创建视图的查询（例如CREATE VIEW...）。 / p>

请参阅：AWS::Athena::NamedQuery - AWS CloudFormation

更新：@Theo指出，AWS CloudFormation还具有AWS Glue功能，其中包括：

AWS :: Glue :: Table

这显然可以用来创建视图。请参阅下面的评论。

使用Cloud Formation模板创建AWS Athena View

3 个答案: