如何通过AWS运行我的map-reduce程序?

时间:2019-07-15 11:33:20

标签: java amazon-web-services maven hadoop mapreduce

我正在尝试使用Hadoop框架运行map-reduce程序。 我需要在AmazonElasticMapReduce实例上运行该程序,但我不断收到以下错误:

  

线程“ main”中的异常java.lang.NoSuchMethodError:com.amazonaws.transform.JsonErrorUnmarshaller:方法(Ljava / lang / Class;)V未找到

我试图修复pom文件并添加AWS开发工具包,更改其版本并分别添加核心。

我的pom.xml文件:

   <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>1.2.1</version>
    </dependency>

    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
        <version>2.7.3</version>
    </dependency>

    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>2.7.3</version>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-ec2</artifactId>
        <version>1.10.2</version>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-s3</artifactId>
        <version>1.10.5</version>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-emr</artifactId>
        <version>1.9.0</version>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk</artifactId>
        <version>1.11.5</version>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-core</artifactId>
        <version>1.11.5</version>
    </dependency>

我正在尝试创建实例:


AWSCredentials creds = new PropertiesCredentials(new FileInputStream(propertiesFilePath));

AmazonElasticMapReduce mapReduce = new AmazonElasticMapReduceClient(creds);

我希望程序可以在AWS实例上运行(并且我的代码中可能会出现更多错误,我可以调试)。

0 个答案:

没有答案