Question

我尝试将spark-cassandra-connector从1.4升级到1.5。

一切似乎都很好，但是当我运行测试用例时，它会在进程之间停留并记录一些错误消息：

线程中的异常＆＃34; dag-scheduler-event-loop＆＃34; java.lang.NoClassDefFoundError：org / apache / spark / sql / DataFrame

我的pom文件如下：

<dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
    <!-- https://mvnrepository.com/artifact/com.datastax.spark/spark-cassandra-connector_2.10 -->
<dependency>
    <groupId>com.datastax.spark</groupId>
    <artifactId>spark-cassandra-connector_2.10</artifactId>
    <version>1.5.0</version>
    </dependency>
        <dependency>
        <groupId>com.google.guava</groupId>
        <artifactId>guava</artifactId>
        <version>16.0.1</version>
    </dependency>
    <!-- Scala Library -->
    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>2.10.5</version>
    </dependency>
    <!--Spark Cassandra Connector-->
    <dependency>
        <groupId>com.datastax.spark</groupId>
        <artifactId>spark-cassandra-connector_2.10</artifactId>
        <version>1.5.0</version>
    </dependency>
    <dependency>
        <groupId>com.datastax.spark</groupId>
        <artifactId>spark-cassandra-connector-java_2.10</artifactId>
        <version>1.5.0</version>
    </dependency>
    <dependency>
      <groupId>com.datastax.cassandra</groupId>
      <artifactId>cassandra-driver-core</artifactId>
      <version>3.0.2</version>
    </dependency>
    <!--Spark-->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.10</artifactId>
        <version>1.5.0</version>
        <exclusions>
          <exclusion>
            <groupId>net.java.dev.jets3t</groupId>
            <artifactId>jets3t</artifactId>
          </exclusion>
        </exclusions>
    </dependency>
  </dependencies>
</project>

提前谢谢!!

任何人都可以帮我这个吗？如果您需要更多信息，请告诉我!!

Answer 1

尝试添加依赖

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.10</artifactId>
    <version>${spark.version}</version>
    <scope>provided</scope>
</dependency>

还要确保您的版本spark-cassandra-connector与您正在使用的Spark版本兼容。当尝试将较旧的spark-cassandra-connector用于较新的Spark版本时，即使具有所有正确的依赖关系，我也会收到相同的错误消息。请参阅此表：https://github.com/datastax/spark-cassandra-connector#version-compatibility

NoClassDefFoundError：spark-cassandra-connector中的org / apache / spark / sql / DataFrame

1 个答案: