Question

我正在尝试在AWS中安排我的python脚本，但是我不希望实例一直在运行。因此，尝试自动化过程：

在特定时间启动EC2实例
在其中运行python脚本
作业完成后停止EC2实例。

我不能直接将此脚本作为Lambda函数运行，因为脚本执行一些需要更多RAM的并行处理，因此选择更大的AWS实例而不是将其编写为lambda函数。此外，不希望此实例一直在运行，因为它很昂贵。

到目前为止，我跟着Automatic starting and stopping of AWS EC2 instances with Lambda and CloudWatch · matoski.com并创建了一个Lambda函数来在特定时间启动和停止实例，但是一旦实例启动，我就找不到运行python脚本的方法。 / p>

有人能指出我正确的方向吗？

Answer 1

我的应用程序每天运行一个实例@ 13:39 UST，并在处理完成后自行关闭。它使用下面的

使用云监视事件规则的预定lambda函数

Cloud watch Event/rules config

lambda触发器将启动一个实例（带有硬编码的id）

import boto3
def lambda_handler(event, context):
    ec2 = boto3.client('ec2', region_name='ap-south-1')
    ec2.start_instances(InstanceIds=['i-xxxxxxx'])
    print('started your instances: ' + str('i-xxxxxx'))
    return

这会触发一个运行cron以执行Python脚本的实例

@reboot python /home/Init.py
脚本完成后，python作业将使用以下代码段自行关闭

import boto.ec2
import boto.utils
import logging
logger=logging.getLogger()
def stop_ec2():
    conn = boto.ec2.connect_to_region("ap-south-1") # or your region
    # Get the current instance's id
    my_id = boto.utils.get_instance_metadata()['instance-id']
    logger.info(' stopping EC2 :'+str(my_id))
    conn.stop_instances(instance_ids=[my_id])

Answer 2

对于将来遇到此问题的开发人员，一种较新的方法是：

使用包含AmazonEC2RoleforSSM策略的角色来创建EC2
创建一个lambda来执行唤醒，运行命令，关机
使用Cloudwatch事件触发lambda

所以：

按照此处的步骤操作：https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
使用以下lambda框架：

import time
import boto3

REGION_NAME = 'us-east-1'

WORKING_DIRECTORY = '<YOUR WORKING DIRECTORY, IF ANY>'

COMMAND = """
    echo "Hello, world!"
    """

INSTANCE_ID = '<YOUR INSTANCE ID>'


def start_ec2():
    ec2 = boto3.client('ec2', region_name=REGION_NAME)
    ec2.start_instances(InstanceIds=[INSTANCE_ID])

    while True:
        response = ec2.describe_instance_status(InstanceIds=[INSTANCE_ID], IncludeAllInstances=True)
        state = response['InstanceStatuses'][0]['InstanceState']

        print(f"Status: {state['Code']} - {state['Name']}")

        # If status is 16 ('running'), then proceed, else, wait 5 seconds and try again
        if state['Code'] == 16:
            break
        else:
            time.sleep(5)

    print('EC2 started')


def stop_ec2():
    ec2 = boto3.client('ec2', region_name=REGION_NAME)
    ec2.stop_instances(InstanceIds=[INSTANCE_ID])

    while True:
        response = ec2.describe_instance_status(InstanceIds=[INSTANCE_ID], IncludeAllInstances=True)
        state = response['InstanceStatuses'][0]['InstanceState']

        print(f"Status: {state['Code']} - {state['Name']}")

        # If status is 80 ('stopped'), then proceed, else wait 5 seconds and try again
        if state['Code'] == 80:
            break
        else:
            time.sleep(5)

    print('Instance stopped')


def run_command():
    client = boto3.client('ssm', region_name=REGION_NAME)

    time.sleep(10)  # I had to wait 10 seconds to "send_command" find my instance 

    cmd_response = client.send_command(
        InstanceIds=[INSTANCE_ID],
        DocumentName='AWS-RunShellScript',
        DocumentVersion="1",
        TimeoutSeconds=300,
        MaxConcurrency="1",
        CloudWatchOutputConfig={'CloudWatchOutputEnabled': True},
        Parameters={
            'commands': [COMMAND],
            'executionTimeout': ["300"],
            'workingDirectory': [WORKING_DIRECTORY],
        },
    )

    command_id = cmd_response['Command']['CommandId']
    time.sleep(1)  # Again, I had to wait 1s to get_command_invocation recognises my command_id

    retcode = -1
    while True:
        output = client.get_command_invocation(
            CommandId=command_id,
            InstanceId=INSTANCE_ID,
        )

        # If the ResponseCode is -1, the command is still running, so wait 5 seconds and try again
        retcode = output['ResponseCode']
        if retcode != -1:
            print('Status: ', output['Status'])
            print('StdOut: ', output['StandardOutputContent'])
            print('StdErr: ', output['StandardErrorContent'])
            break

        print('Status: ', retcode)
        time.sleep(5)

    print('Command finished successfully') # Actually, 0 means success, anything else means a fail, but it didn't matter to me
    return retcode


def lambda_handler(event, context):
    retcode = -1
    try:
        start_ec2()
        retcode = run_command()
    finally:  # Independently of what happens, try to shutdown the EC2
        stop_ec2()

    return retcode

按照此处的步骤操作：https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/RunLambdaSchedule.html

Answer 3

我在使用本文中的解决方案启动和停止实例时遇到了问题。然后，我按照https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/上的说明进行操作，这确实很容易。基本上：

转到https://console.aws.amazon.com/iam/home#/home，然后在左侧，单击“策略”，然后单击“创建策略”。然后单击“ JSON”选项卡。然后复制粘贴此内容以创建新策略：

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:Start*",
        "ec2:Stop*"
      ],
      "Resource": "*"
    }
  ]
}

转到https://console.aws.amazon.com/iam/home#/home，然后在左侧选择“角色”。确保选择Lambda作为AWS服务，并附加在步骤1中创建的策略。
然后转到Lambda控制台，单击“创建函数”。选择Python 3.7，然后单击“权限”和“使用现有角色”旁边的下拉列表，并附加在步骤2中创建的IAM角色。
使用此代码作为您的代码：

import boto3
region = 'us-west-1' # Dont use the specific, like instead of us-east-1d just write us-east-1
instances = ['i-xxxxxxxxxxxx']
ec2 = boto3.client('ec2', region_name=region)

def lambda_handler(event, context):
    ec2.start_instances(InstanceIds=instances)
    print('started your instances: ' + str(instances))

启动您的EC2实例，然后键入which python来找到python的路径并将其记录下来。然后，键入crontab -e以编辑您的CRON作业。不要使用sudo ...因为有时候sudo可能会在您没有使用它来运行Python文件时弄乱它们。在我的实例中，我有一个pgpass文件，其中存储了sudo无法看到的密码，但是删除sudo可以成功！
在crontab编辑器中的注释行之后，键入@reboot /path/to/python /path/to/file.py。例如，对我来说，这是@reboot /home/init/python /home/init/Notebooks/mypredictor.py
在Python文件的末尾，您需要停止实例。您可以这样做：

import boto3
region = 'us-west-1' # Dont use the specific, like instead of us-east-1d just write us-east-1
instances = ['i-xxxxxxxxxxxx']
ec2 = boto3.client('ec2', region_name=region)

ec2.stop_instances(InstanceIds=instances)

安排启动EC2实例并在其中运行python脚本

3 个答案: