如何使用Eclipse在Amazon EMR中运行Hadoop作业?

时间:2015-11-29 11:49:01

标签: eclipse hadoop amazon-web-services emr

我已按照亚马逊here提供的教程,但似乎我的代码无法运行。

我得到的错误:

Exception in thread "main" java.lang.Error: Unresolved compilation problem: 
    The method withJobFlowRole(String) is undefined for the type AddJobFlowStepsRequest

    at main.main(main.java:38)

我的完整代码:

import java.io.IOException;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.PropertiesCredentials;
import com.amazonaws.services.elasticmapreduce.*;
import com.amazonaws.services.elasticmapreduce.model.AddJobFlowStepsRequest;
import com.amazonaws.services.elasticmapreduce.model.AddJobFlowStepsResult;
import com.amazonaws.services.elasticmapreduce.model.HadoopJarStepConfig;
import com.amazonaws.services.elasticmapreduce.model.StepConfig;
import com.amazonaws.services.elasticmapreduce.util.StepFactory;

public class main {

    public static void main(String[] args) {

        AWSCredentials credentials = null;
        try {
            credentials = new PropertiesCredentials(
                main.class.getResourceAsStream("AwsCredentials.properties"));
        } catch (IOException e1) {
            System.out.println("Credentials were not properly entered into AwsCredentials.properties.");
            System.out.println(e1.getMessage());
            System.exit(-1);
        }

        AmazonElasticMapReduce client = new AmazonElasticMapReduceClient(credentials);

        // predefined steps. See StepFactory for list of predefined steps
        StepConfig hive = new StepConfig("Hive", new StepFactory().newInstallHiveStep());

        // A custom step
        HadoopJarStepConfig hadoopConfig1 = new HadoopJarStepConfig()
            .withJar("s3://mywordcountbuckett/binary/WordCount.jar")
            .withMainClass("com.my.Main1") // optional main class, this can be omitted if jar above has a manifest
            .withArgs("--verbose"); // optional list of arguments
        StepConfig customStep = new StepConfig("Step1", hadoopConfig1);

        AddJobFlowStepsResult result = client.addJobFlowSteps(new AddJobFlowStepsRequest()  
            .withJobFlowRole("jobflow_role")
            .withServiceRole("service_role")
            .withSteps(hive, customStep));
        System.out.println(result.getStepIds());

    }
}

代码没有运行的原因是什么? 有没有基于最新版本的教程?

0 个答案:

没有答案