Cloudera VM上的Apache Giraph - org.apache.hadoop的POM：hadoop-core：jar：2.6.0缺失，没有依赖项信息

时间：2017-12-08 19:44:58

标签： maven hadoop pom.xml cloudera-quickstart-vm giraph

我是Hadoop / Giraph和Java的新手。作为任务的一部分，我在其上下载了Cloudera Quickstart VM和Giraph。我正在使用这本名为＆＃34;实用图形分析与Apache Giraph的书;作者：Shaposhnik，Roman，Martella，Claudio，Logothetis，Dionysios＆＃34;我试图在第111页（Twitter追随者图）上运行第一个例子。

尝试使用群集2.6.0-mr1-cdh5.12.0上的hadoop版本运行已更改的pom.xml文件时，请找到以下错误

`[cloudera@quickstart first]$ mvn clean install
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building book-examples 1.0.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ book-examples ---
[INFO] Deleting /home/cloudera/workspace/first/target
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ book-examples ---
[debug] execute contextualize
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 0 resource
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ book-examples ---
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 1 source file to /home/cloudera/workspace/first/target/classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[5,27] error: package org.apache.hadoop.io does not exist
[ERROR] /home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[6,27] error: package org.apache.hadoop.io does not exist
[ERROR] /home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[7,29] error: cannot find symbol
[ERROR]  package org.apache.hadoop.util
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[14,17] error: cannot find symbol
[ERROR]  class IntWritable
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[14,30] error: cannot find symbol
[ERROR]  class IntWritable
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[15,0] error: cannot find symbol
[ERROR]  class NullWritable
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[15,14] error: cannot find symbol
[ERROR]  class NullWritable
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[17,28] error: cannot find symbol
[ERROR]  class GiraphHelloWorld
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[18,3] error: cannot find symbol
[ERROR]  class GiraphHelloWorld
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[18,16] error: cannot find symbol
[ERROR]  class GiraphHelloWorld
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[19,12] error: cannot find symbol
[ERROR]  class GiraphHelloWorld
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[24,12] error: cannot find symbol
[ERROR]  class GiraphHelloWorld
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[24,25] error: cannot find symbol
[ERROR]  class GiraphHelloWorld
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[34,14] error: cannot find symbol
[INFO] 14 errors 
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.495s
[INFO] Finished at: Fri Dec 08 14:57:01 PST 2017
[INFO] Final Memory: 18M/57M

`

我根据堆栈溢出响应添加了Cloudera Repository。请找到适用于上述错误的更新的pom xml：

`<?xml version="1.0" encoding="UTF-8"?>
<project>
<modelVersion>4.0.0</modelVersion>

<groupId>giraph</groupId>
<artifactId>book-examples</artifactId>
<version>1.0.0</version>

<dependencies>
<dependency>
<groupId>org.apache.giraph</groupId>
<artifactId>giraph-core</artifactId>
<version>1.1.0</version>
</dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>2.6.0-mr1-cdh5.12.0</version>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.4</version>
                <executions>
                    <execution>
                        <id>create-jar-bundle</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                        <configuration>
                            <descriptorRefs>
                                <descriptorRef>jar-with-dependencies</descriptorRef>
                            </descriptorRefs>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
</plugins>
</build>

<repositories>
        <repository>
            <id>cloudera</id>
            <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
    </repositories>

</project>`

hadoop书中的版本是1.2.1。两个版本之间存在书籍依赖关系的问题。

如果有人帮助我理解如何处理这个错误，那就太好了。

提前致谢。

1 个答案:

答案 0 :(得分：1)

您图书副本中的pom.xml已过时。请改用这个。来源：Github上的book examples repository。

编辑：

您希望使用最新版本的hadoop-core，但最新的Maven Central Repository（默认存储库）提供的是1.2.1。您需要使用Cloudera Repository来获取最新版本的库。为此，只需将存储库添加到pom.xml：

<?xml version="1.0" encoding="UTF-8"?>
<project>
    <modelVersion>4.0.0</modelVersion>
    <groupId>giraph</groupId>
    <artifactId>book-examples</artifactId>
    <version>1.0.0</version>

    <dependencies>
        <dependency>
            <groupId>org.apache.giraph</groupId>
            <artifactId>giraph-core</artifactId>
            <version>1.1.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-core</artifactId>
            <version>2.6.0-mr1-cdh5.12.0</version>
        </dependency>
    </dependencies>

    <build>
    </build>

    <repositories>
        <repository>
            <id>cloudera</id>
            <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
    </repositories>
</project>

你现在应该看到Maven首先尝试在Cloudera找到罐子，然后回到中央：

$ mvn clean install
[INFO] Scanning for projects...
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Building book-examples 1.0.0
[INFO] ------------------------------------------------------------------------
...
Downloading: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-core/2.6.0-mr1-cdh5.12.0/hadoop-core-2.6.0-mr1-cdh5.12.0.pom
Downloaded: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-core/2.6.0-mr1-cdh5.12.0/hadoop-core-2.6.0-mr1-cdh5.12.0.pom (6.4 kB at 2.9 kB/s)
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.661 s
[INFO] Finished at: 2017-12-08T22:56:04+01:00
[INFO] Final Memory: 15M/224M
[INFO] ------------------------------------------------------------------------

编辑2：

好的，我终于明白了。从版本2开始，Hadoop改变了它的包装，因此不应声明对hadoop-core的依赖，而应使用hadoop-client，这是一个为您聚合所有必要依赖项的元数据包。从您的hadoop-core中移除pom.xml相关性，然后添加：

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>2.9.0</version>
</dependency>
相关问题
最新问题