通过SDK启用AWS EMRFS Consistent View

Question

通常通过emrfs-site.xml启用emrfs一致性 http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emrfs-configure-consistent-view.html

有谁知道是否可以通过SDK访问这些设置？

Answer 1

是的，您有完整的文档： http://docs.aws.amazon.com/ElasticMapReduce/latest/API/Welcome.html

您需要首先授权与AWS的连接，而不是使用API​​根据需要配置应用程序。 看这里： http://docs.aws.amazon.com/ElasticMapReduce/latest/API/CommonParameters.html http://docs.aws.amazon.com/ElasticMapReduce/latest/API/EmrConfigurations.html

Answer 2

要使用Java SDK启用EMRFS，需要将“ emrfs-site”配置添加到RunJobFlowRequest，并且必须将 fs.s3.consistent 属性设置为 true 。像这样：

Map<String, String> emrfsProperties = new HashMap<>();
emrfsProperties.put("fs.s3.consistent", "true");

RunJobFlowRequest request = new RunJobFlowRequest()
        ....
        .withServiceRole(SERVICE_ROLE)
        .withJobFlowRole(JOB_FLOW_ROLE)
        .withConfigurations(
                new Configuration().withClassification("yarn-site").withProperties(yarnProperties),
                new Configuration().withClassification("emrfs-site").withProperties(emrfsProperties)
        )
        .withInstances(new JobFlowInstancesConfig()
                .withEc2KeyName(EC2_KEYPAIR)
                ....

可以找到here

的EMRFS配置参数的完整列表