Question

我当时正在使用Java SDK（下面是一个代码段）启动一个AWS ERM集群，效果很好。

BasicAWSCredentials awsCreds = new BasicAWSCredentials(accessKeyId, secretAccessKeyId);
AmazonElasticMapReduce emrClient = AmazonElasticMapReduceClientBuilder.standard()
                    .withCredentials(new AWSStaticCredentialsProvider(awsCreds))
                    .withRegion(region)
                    .build();

JobFlowInstancesConfig jobFlowInstanceConfig = new JobFlowInstancesConfig()
                .withEc2SubnetId("subnetId")
                .withEc2KeyName("ec2KeyName") 
                .withInstanceCount(3) 
                .withKeepJobFlowAliveWhenNoSteps(true)    
                .withMasterInstanceType(c5.4xlarge)
                .withSlaveInstanceType(c5.4xlarge); 


        // create the cluster
        RunJobFlowRequest request = new RunJobFlowRequest()
                .withName("clusterName")
                .withReleaseLabel("emr-5.23.0")
                .withApplications("<Added following in applications Hadoop,Spark,Ganglia,Zeppelin>")
                .withLogUri("s3 path")
                .withServiceRole("EMR_DefaultRole")
                .withJobFlowRole("EMR_EC2_DefaultRole")
                .withInstances(jobFlowInstanceConfig);

RunJobFlowResult runJobFlowResult = emrClient.runJobFlow(request);

稍后在另一个AWS环境中，我们的AWS团队创建了一个角色，以从特定的EC2实例创建集群。但是我无法创建集群。以下是具有其他配置的代码段，以及与之前的配置有关的更改。

没有accessKeyId和secretAccessKeyId
EMR_EC2_DefaultRole更改为已配置的角色

安全配置已添加

AmazonElasticMapReduce emrClient = AmazonElasticMapReduceClientBuilder.standard()
                .withRegion(region)
                .build();

JobFlowInstancesConfig jobFlowInstanceConfig = new JobFlowInstancesConfig()
            .withEc2SubnetId("subnetId")
            .withEc2KeyName("ec2KeyName") 
            .withInstanceCount(3) 
            .withKeepJobFlowAliveWhenNoSteps(true)    
            .withMasterInstanceType(c5.4xlarge)
            .withSlaveInstanceType(c5.4xlarge); 

RunJobFlowRequest request = new RunJobFlowRequest()
            .withName("clusterName")
            .withReleaseLabel("emr-5.23.0")
            .withApplications("<Added following in applications Hadoop,Spark,Ganglia,Zeppelin>")
            .withLogUri("s3 path")
            .withServiceRole("EMR_DefaultRole")
            .withJobFlowRole("name-of-role-created")
            .withInstances(jobFlowInstanceConfig)
            .withSecurityConfiguration("Security configuration Name");

RunJobFlowResult runJobFlowResult = emrClient.runJobFlow(request);

我收到以下错误：

com.amazonaws.services.elasticmapreduce.model.AmazonElasticMapReduceException: Role '' is not well-formed. (Service: AmazonElasticMapReduce; Status Code: 400; Error Code: ValidationException; Request ID: 0d5ed77e-ed0e-49fd-bd33-f88213ce08c3)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1701)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1356)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1102)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:759)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:733)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:715)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:675)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:657)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:521)
    at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.doInvoke(AmazonElasticMapReduceClient.java:2043)
    at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.invoke(AmazonElasticMapReduceClient.java:2010)
    at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.invoke(AmazonElasticMapReduceClient.java:1999)
    at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.executeRunJobFlow(AmazonElasticMapReduceClient.java:1770)
    at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.runJobFlow(AmazonElasticMapReduceClient.java:1742)

由于上述错误表明角色的格式不正确，因此我尝试使用不同的格式仍然遇到相同的问题。以下是我在.withJobFlowRole（“角色创建的名称”）中添加的不同格式

arn:aws:iam::639116131780:role/name-of-role-created
arn:aws:iam::639116131780:instance-profile/name-of-role-created
arn:aws:iam::639116131780:role/name-of-role-created/*
arn:aws:iam::639116131780:instance-profile/name-of-role-created/*
arn:aws:sts::639116131780:assumed-role/name-of-role-created
arn:aws:sts::639116131780:assumed-role/name-of-role-created/*

我每次都会遇到相同的错误。

com.amazonaws.services.elasticmapreduce.model.AmazonElasticMapReduceException: Role 'arn:aws:iam::639116131780:role/name-of-role-created' is not well-formed. (Service: AmazonElasticMapReduce; Status Code: 400; Error Code: ValidationException; Request ID: 0d5ed77e-ed0e-49fd-bd33-f88213ce08c3)

Answer 1

根据docs，JobFlowRole参数不是ARN，而只是一个字符串，如EMR_EC2_DefaultRole（默认值）。使用这样的格式。

Answer 2

JobFlowRole是应用于EMR实例的角色，不是创建EMR时要使用的角色。我认为您误读了该选项。

如果要应用不使用API密钥的角色，则必须挖掘AWS凭证。例如在S3中，

S3Client s3 = S3Client.builder()
              .credentialsProvider(InstanceProfileCredentialsProvider.builder().build())
              .build();

其中

InstanceProfileCredentialsProvider.builder().build()

使用实例的角色。

无法使用Java SDK创建Aws Emr群集

2 个答案: