Question

我正在尝试创建一个spark应用程序来运行来自Cassandra DSE的spark sql查询，但我无法使用maven构建我的应用程序。附加到我的代码和pom.xml下面。

package com.spark
import com.datastax.spark.connector._
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.sql._
import org.apache.spark.sql.SQLContext

object controller {
def main(args : Array[String]){
val conf = new SparkConf().setMaster("local").setAppName("testing").
set("spark.cassandra.connection.host","192.168.0.113").
set("spark.driver.allowMultipleContexts","true")

val sc = new SparkContext(conf) 
val sqlc = new org.apache.spark.sql.SQLContext(sc)
print(sqlc.sql("select * from test.test limit 10).collect().toList)"));
 }
}

POM.xml如下： -

<project xmlns="http://maven.apache.org/POM/4.0.0" 
  xmlns:xsi= "http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" >
 < modelVersion >4.0.0< /modelVersion>

< groupId>com.spark< /groupId>
<artifactId>Controller</artifactId>
<version>0.0.1-SNAPSHOT</version>
<pluginRepositories>
    <pluginRepository>
        <id>scala-tools.org</id>
        <name>Scala-tools Maven2 Repository</name>
        <url>http://scala-tools.org/repo-releases</url>
    </pluginRepository>
</pluginRepositories>
<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.10</artifactId>
        <version>1.4.0</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/com.datastax.spark/spark-cassandra-connector -->
    <dependency>
        <groupId>com.datastax.spark</groupId>
        <artifactId>spark-cassandra-connector_2.11</artifactId>
        <version>2.0.7</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>2.2.0</version>
    </dependency>
</dependencies>
<build>
<plugins>
        <!-- mixed scala/java compile -->

        <plugin>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <source>1.7</source>
                <target>1.7</target>
            </configuration>
        </plugin>
        <!-- for fatjar -->
        <plugin>
            <!-- <groupId>org.apache.maven.plugins</groupId> -->
            <artifactId>maven-assembly-plugin</artifactId>
            <!-- <version>2.4</version> -->
            <!-- <dependencies> <dependency> <groupId>src.main.resources.assemblies</groupId> 
                <artifactId>myassembly</artifactId> <version>1.0-SNAPSHOT</version> </dependency> 
                </dependencies> -->
            <configuration>
                <!-- <finalName>test</finalName> -->
                <archive>
                    <manifest>
                        <mainClass>src.main.com.spark.Controller</mainClass>
                    </manifest>
                </archive>
                <descriptorRefs>
                    <descriptorRef>jar-with-dependencies</descriptorRef>
                </descriptorRefs>
                <!-- <descriptors>
                    <descriptor>src\main\resources\assemblies\myassembly.xml</descriptor>
                </descriptors> -->
                <!-- <appendAssemblyId>false</appendAssemblyId> -->
            </configuration>
            <executions>
                <execution>
                    <id>make-assembly</id>
                    <phase>package</phase>
                    <goals>
                        <goal>single</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-jar-plugin</artifactId>
            <configuration>
                <archive>
                    <manifest>
                        <addClasspath>true</addClasspath>
                         <mainClass>fully.qualified.MainClass</mainClass>
                    </manifest>
                </archive>
            </configuration>
        </plugin>
    </plugins>
    <pluginManagement>
        <plugins>
            <!--This plugin's configuration is used to store Eclipse m2e settings 
                only. It has no influence on the Maven build itself. -->
            <plugin>
                <groupId>org.eclipse.m2e</groupId>
                <artifactId>lifecycle-mapping</artifactId>
                <version>1.0.0</version>
                <configuration>
                    <lifecycleMappingMetadata>
                        <pluginExecutions>
                            <pluginExecution>
                                <pluginExecutionFilter>
                                    <groupId>org.scala-tools</groupId>
                                    <artifactId> maven-scala-plugin                                     </artifactId>
                                    <versionRange> [2.15.2,)                                        </versionRange>
                                    <goals>
                                        <goal>compile</goal>
                                        <goal>testCompile</goal>
                                    </goals>
                                </pluginExecutionFilter>
                                <action>
                                    <execute />
                                </action>
                            </pluginExecution>
                        </pluginExecutions>
                    </lifecycleMappingMetadata>
                </configuration>
            </plugin>
        </plugins>
    </pluginManagement>
</build>
</project>

在清洁安装时，它会出现以下错误：- Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.2-beta-5:single (make-assembly) on project Controller: Failed to create assembly: Error creating assembly archive jar-with-dependencies: invalid CEN header (bad signature) -> [Help 1] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

Answer 1

看起来~/.m2/repository/文件夹中的jar已损坏。您需要删除完整目录，或仅删除当前使用的依赖项...

获取新错误消息后更新：项目中使用了不同版本的Scala - spark-core使用2.10构建，而spark-sql＆amp; cassandra连接器在2.11上。

Cassandra DSE Spark scala应用程序maven构建错误

1 个答案: