Question

我的工作是在Hadoop的地图中将每条记录写入DynamoDB。

我无法使用具有httpclient-4.2.5.jar和httpcore-4.2.5.jar的Hadoop 2.6。

我使用的AWS是使用httpclient-4.5.2.jar和httpcore-4.4.4.jar构建的。

当我使用classpath包含新的jar文件时，它会给出以下异常。

java.lang.Exception: java.lang.NoSuchFieldError: INSTANCE
    at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.lang.NoSuchFieldError: INSTANCE
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.<clinit>(SSLConnectionSocketFactory.java:144)
    at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.getPreferredSocketFactory(ApacheConnectionManagerFactory.java:87)
    at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:65)
    at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:58)
    at com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(ApacheHttpClientFactory.java:49)

对我而言，看起来Hadoop是使用旧库构建的，并且语法中的某些内容已经发生了变化。

比重新编译旧的AWS源更合理的解决方案是什么？

Answer 1

作为更新，我不得不切换到Maven并稍微使用版本。

 <!-- http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-common</artifactId>
    <version>2.6.0</version>
</dependency>

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-mapreduce-client-core</artifactId>
    <version>2.6.0</version>
</dependency>

 <!-- http://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk -->
 <dependency>
      <groupId>org.apache.httpcomponents</groupId>
      <artifactId>httpclient</artifactId>
      <version>4.3.4</version>
      <scope>compile</scope>
    </dependency>

<dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-dynamodb</artifactId>
        <version>1.9.13</version>
    </dependency>

    <dependency>
        <groupId>org.netpreserve.commons</groupId>
        <artifactId>webarchive-commons</artifactId>
        <version>1.1.4</version>
    </dependency>

最后，它有效

Hadoop 2.6.x和Amazon AWS SDK库冲突了http-core冲突

1 个答案: