Question

我正在尝试在Apache Spark中的单个集群上运行PageRank示例。它使用了Iterables类的com.google.common.collect包。我已经在我的pom.xml

中添加了maven依赖项

<dependency>
        <groupId>com.google.collections</groupId>
        <artifactId>google-collections</artifactId>
        <version>1.0</version>
</dependency>

该项目在Eclipse中构建，并向Maven依赖项添加了一个jar。

Maven dependency

我正在使用以下Maven插件

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.3</version>
        </plugin>
    </plugins>
</build>

但是当我将作业提交给Spark时，它会在NoClassDefFoundError

上出错

Caused by: java.lang.NoClassDefFoundError: com/google/common/collect/Iterables
    at spark.JavaPageRank$3.call(JavaPageRank.java:108)
    at spark.JavaPageRank$3.call(JavaPageRank.java:1)
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:146)
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:146)
    at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
    at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:209)
    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:73)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.google.common.collect.Iterables
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

这里有什么建议吗？我尝试了几种解决方案，包括以下内容，但无济于事。

Java ClassNotFoundException with maven dependency

尽管添加了Maven依赖项，但ClassNotFoundException仍然存在

0 个答案: