Question

我正在运行spark 1.2.1。我正在尝试使用spark sql在java程序中读取一个hive表。我还确保在类路径中可以使用spark-core和spark-sql jar。执行以下语句时，我得到NoSuchMethodError和程序中断：

    import org.apache.spark.sql.api.java.JavaSQLContext;
.
.
    JavaSQLContext sqlContext = new JavaSQLContext(sc);

java.lang.NoSuchMethodError：org.apache.spark.sql.SQLContext。（Lorg / apache / spark / api / java / JavaSparkContext;）v

请在这方面指导我。

构建Maven脚本

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>org.xyz</groupId>
    <artifactId>updateprocess</artifactId>
    <version>0.0.1-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.10</artifactId>
            <version>1.2.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.10</artifactId>
            <version>1.2.1</version>
        </dependency>
    <dependencies>
</project>

我确实在本地模式和纱线上运行它。这两个时间都有同样的错误。

echo $ LIB_JARS 依赖性/火花core_2.10-1.2.1.jar，依赖/火花sql_2.10-1.2.1.jar

spark-submit --deploy-mode client --master yarn-client --class org.xyz.ExtractTestData  --jars $LIB_JARS upda*.jar input output/Account.txt output/Customer.txt

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.SQLContext.<init>(Lorg/apache/spark/api/java/JavaSparkContext;)V
        at org.xyz.ExtractTestData.main(ExtractTestData.java:58)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

java.lang.NoSuchMethodError：org.apache.spark.sql.SQLContext。<init>（Lorg / apache / spark / api / java / JavaSparkContext

构建Maven脚本

0 个答案: