Question

我已经在AWS lambda中创建了一个函数，如下所示：

import boto3
import numpy as np
import pandas as pd
import s3fs
from io import StringIO


def test(event=None, context=None):
    # creating a pandas dataframe from an api
    # placing 2 csv files in S3 bucket

此函数查询外部API，并将2个csv文件放在S3存储桶中。我想在Airflow中触发此功能，我找到了以下代码：

import boto3, json, typing

def invokeLambdaFunction(*, functionName:str=None, payload:typing.Mapping[str, str]=None):
    if  functionName == None:
        raise Exception('ERROR: functionName parameter cannot be NULL')
    payloadStr = json.dumps(payload)
    payloadBytesArr = bytes(payloadStr, encoding='utf8')
    client = boto3.client('lambda')
    response = client.invoke(
        FunctionName=test,
        InvocationType="RequestResponse",
        Payload=payloadBytesArr
    )
    return response



if __name__ == '__main__':
    payloadObj = {"something" : "1111111-222222-333333-bba8-1111111"}
    response = invokeLambdaFunction(functionName='test',  payload=payloadObj)
    print(f'response:{response}')

但是据我了解，此代码段未连接到S3。这是从Airflow触发AWS Lambda功能的正确方法，还是有更好的方法？

Answer 1

我建议您使用AwsLambdaHook：

https://airflow.apache.org/docs/stable/_api/airflow/contrib/hooks/aws_lambda_hook/index.html#module-airflow.contrib.hooks.aws_lambda_hook

您可以检查一个显示其用法来触发lambda函数的测试：

https://github.com/apache/airflow/blob/master/tests/providers/amazon/aws/hooks/test_lambda_function.py

从Airflow触发AWS Lambda函数

1 个答案: