Question

我想在EMR中自动执行群集创建任务。我有一个json文件其中包含需要在新集群上应用的配置，我想编写一个shell脚本，为我自动执行此任务。

是否可以通过提供json文件中的所有配置来在EMR中创建集群？

例如我有这个文件

{
"Cluster": { 
    "Ec2InstanceAttributes": {
        "EmrManagedMasterSecurityGroup": "sg-00b10b71", 
        "RequestedEc2AvailabilityZones": [], 
        "AdditionalSlaveSecurityGroups": [], 
        "AdditionalMasterSecurityGroups": [], 
        "RequestedEc2SubnetIds": [
            "subnet-02291b3e"
        ], 
        "Ec2SubnetId": "subnet-02291b3e", 
        "IamInstanceProfile": "EMR_EC2_DefaultRole", 
        "Ec2KeyName": "perf_key_pair", 
        "Ec2AvailabilityZone": "us-east-1e", 
        "EmrManagedSlaveSecurityGroup": "sg-f2b30983"
    }, 
    "Name": "NitinJ-Perf", 
    "ServiceRole": "EMR_DefaultRole", 
    "Tags": [
        {
            "Value": "Perf-Nitink", 
            "Key": "Qubole"
        }
    ], 
    "Applications": [
        {
            "Version": "3.7.2", 
            "Name": "Ganglia"
        }, 
        {
            "Version": "2.7.3", 
            "Name": "Hadoop"
        }, 
        {
            "Version": "2.1.1", 
            "Name": "Hive"
        }, 
        {
            "Version": "0.16.0", 
            "Name": "Pig"
        }, 
        {
            "Version": "0.8.4", 
            "Name": "Tez"
        }
    ], 
    "MasterPublicDnsName": "ec2-34-229-254-217.compute-1.amazonaws.com", 
    "ScaleDownBehavior": "TERMINATE_AT_INSTANCE_HOUR", 
    "InstanceGroups": [
        {
            "RequestedInstanceCount": 4, 
            "Status": {
                "Timeline": {
                    "ReadyDateTime": 1499150835.979, 
                    "CreationDateTime": 1499150533.99
                }, 
                "State": "RUNNING", 
                "StateChangeReason": {
                    "Message": ""
                }
            }, 
            "Name": "Core Instance Group", 
            "InstanceGroupType": "CORE", 
            "EbsBlockDevices": [], 
            "ShrinkPolicy": {}, 
            "Id": "ig-34P3CVF8ZL5CW", 
            "Configurations": [], 
            "InstanceType": "r3.4xlarge", 
            "Market": "ON_DEMAND", 
            "RunningInstanceCount": 4
        }, 
        {
            "RequestedInstanceCount": 1, 
            "Status": {
                "Timeline": {
                    "ReadyDateTime": 1499150804.591, 
                    "CreationDateTime": 1499150533.99
                }, 
                "State": "RUNNING", 
                "StateChangeReason": {
                    "Message": ""
                }
            }, 
            "Name": "Master Instance Group", 
            "InstanceGroupType": "MASTER", 
            "EbsBlockDevices": [], 
            "ShrinkPolicy": {}, 
            "Id": "ig-3V7EHQ36187PY", 
            "Configurations": [], 
            "InstanceType": "r3.4xlarge", 
            "Market": "ON_DEMAND", 
            "RunningInstanceCount": 1
        }
    ], 
    "Configurations": [
        {
            "Properties": {
                "hive.vectorized.execution.enabled": "true"
            }, 
            "Classification": "hive-site"
        }
    ]
}
}

我可以使用某些命令（如

）在EMR上创建群集

aws emr create-cluster --cli-input-json file://'pwd'/emr_cluster_up.json

Answer 1

根据AWS CLI文档，AWS CLI没有此类选项。但是，如果要使用JSON文件自动创建EMR群集。您可以使用云编队并自动创建群集。

Getting Started with AWS CloudFormation

是否可以通过提供json文件中的所有配置在EMR中创建集群

1 个答案: