Question

我正在尝试在java上创建EMR集群，但我无法在EMR集群列表中找到它，也无法在EC2上看到请求的实例。

EMR角色确实存在：

sqlInjection@VirtualBox:~$ aws iam list-roles | grep EMR
            "RoleName": "EMR_DefaultRole", 
            "Arn": "arn:aws:iam::removed:role/EMR_DefaultRole"
            "RoleName": "EMR_EC2_DefaultRole", 
            "Arn": "arn:aws:iam::removed:role/EMR_EC2_DefaultRole"

现在我的java代码：

AWSCredentials awsCredentials = new BasicAWSCredentials(awsKey, awsKeySecret);
AmazonElasticMapReduce emr = new AmazonElasticMapReduceClient(awsCredentials);

StepFactory stepFactory = new StepFactory();
        StepConfig enabledebugging = new StepConfig()
               .withName("Enable debugging")
               .withActionOnFailure("TERMINATE_JOB_FLOW")
               .withHadoopJarStep(stepFactory.newEnableDebuggingStep());

    HadoopJarStepConfig hadoopConfig1 = new HadoopJarStepConfig()
    .withJar("s3://foo.bucket/hadoop_jar/2015-01-12/foo.jar")
    .withMainClass("com.strackoverflow.DriverFoo") // optional main class, this can be omitted if jar above has a manifest
    .withArgs("--input=s3://foo.bucket/logs/,s3://foo.bucket/morelogs/", "--output=s3://foo.bucket/myEMROutput" , "--inputType=text"); // i have custom java code to handle the --input, --output and --inputType parameters


    StepConfig customStep = new StepConfig("Step1", hadoopConfig1);

    Collection <StepConfig> steps = new ArrayList<StepConfig>();
    {
        steps.add(enabledebugging);
        steps.add(customStep);
    }


    JobFlowInstancesConfig instancesConfig = new JobFlowInstancesConfig()
    .withEc2KeyName("fookey") //not fookey.pem
    .withInstanceCount(2)
    .withKeepJobFlowAliveWhenNoSteps(false) // on aws example is set to true
    .withMasterInstanceType("m1.medium")
    .withSlaveInstanceType("m1.medium");


    RunJobFlowRequest request = new RunJobFlowRequest()
     .withName("java programatic request")
     .withAmiVersion("3.3.1")
     .withSteps(steps) // on the amazon example is lunched debug and hive, here is debug and a jar
     .withLogUri("s3://devel.rui/emr_clusters/pr01/")
     .withInstances(instancesConfig)
     .withVisibleToAllUsers(true);

       RunJobFlowResult result = emr.runJobFlow(request);

       System.out.println("toString "+ result.toString());
       System.out.println("getJobFlowId "+ result.getJobFlowId());
       System.out.println("hashCode "+ result.hashCode());

我的群集在哪里？我无法在群集列表中看到它，未创建输出文件夹，日志文件夹保持为空并且EC2上没有可见的实例。

程序输出此

toString {JobFlowId: j-2xxxxxxU}
getJobFlowId j-2xxxxxU
hashCode -1xxxxx4

我已按照此处的说明创建群集 http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/calling-emr-with-java-sdk.html

这就是创建java作业 http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-common-programming-sample.html

Answer 1

在亚马逊示例中，未配置该区域。

配置区域后，群集正常启动。

    AmazonElasticMapReduce emr = new AmazonElasticMapReduceClient(awsCredentials);
    emr.setRegion(Region.getRegion(Regions.EU_WEST_1));

我的EMR集群在哪里

1 个答案: