我的Maven项目配置错误了什么？不选择依赖项或类结构

Question

我的项目似乎配置有误，导致maven错过了我的基础pom.xml中的依赖项。当我尝试使用intellij的构建选项构建模块，或使用mvn clean软件包构建项目时，出现错误消息“对象apache不是软件包org的成员”。我的pom文件包含以下内容：

<dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_${scala.major.version}.${scala.minor.version}</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
        <exclusions>
          ...
        </exclusions>
      </dependency>

我已经确认本地.m2存储库中存在spark-sql_2.11（我的版本）jar，并且我的设置已配置为自动导入Maven项目。

当我删除对org.apache._的引用时，通过mvn clean package和intellij的build module菜单选项都成功构建了jar，但是两个jar的功能不同。通过intellij（/out/artifacts/[project]_jar/[project].jar）构建模块而创建的jar可以通过spark-submit很好地工作，但是使用从mvn clean软件包创建的jar（/ target / [project]- test-SNAPSHOT.jar）会抛出错误“在jar中未设置主类”，尽管它看起来确实存在，“ jar tvf”显示了该类，而MANIFEST.MF已将其正确列为主类。

在我的scala类和maven项目之间似乎几乎没有联系，但是我不知道在哪里。我们将不胜感激有关如何诊断为什么此方法无效的任何提示，如有必要，我绝对可以提供其他详细信息。预先感谢！

编辑：添加pom.xml的全部内容

    <?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <artifactId>basepom</artifactId>
    <groupId>com.cdtk.maven.basepom</groupId>
    <version>1.0-SNAPSHOT</version>
    <relativePath />
  </parent>

  <groupId>com.[company].testproject2</groupId>
  <artifactId>swtest2</artifactId>
  <version>test-SNAPSHOT</version>
  <packaging>jar</packaging>

  <description>
    This parent pom holds all the necessary version
    and configuration information for all submodules.
  </description>

  <properties>
    <required.maven.version>3.5.2</required.maven.version>
    <upper.maven.version>3.5.999</upper.maven.version>
    <required.java.update>20</required.java.update>

    <scala.major.version>2</scala.major.version>
    <scala.minor.version>11</scala.minor.version>
    <scala.patch.version>0</scala.patch.version>

    <spark.version>2.3.0</spark.version>
  </properties>

  <dependencyManagement>
    <dependencies>
      <!-- =================================== -->
      <!--        Core Spark BOM               -->
      <!-- =================================== -->
      <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_${scala.major.version}.${scala.minor.version}</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
        <exclusions>
          <exclusion>
            <artifactId>commons-beanutils-core</artifactId>
            <groupId>commons-beanutils</groupId>
          </exclusion>
          <exclusion>
            <artifactId>jersey-server</artifactId>
            <groupId>com.sun.jersey</groupId>
          </exclusion>
          <exclusion>
            <artifactId>javax.inject</artifactId>
            <groupId>javax.inject</groupId>
          </exclusion>
          <exclusion>
            <artifactId>javax.ws.rs-api</artifactId>
            <groupId>javax.ws.rs</groupId>
          </exclusion>
          <exclusion>
            <artifactId>stax-api</artifactId>
            <groupId>javax.xml.stream</groupId>
          </exclusion>
          <exclusion>
            <artifactId>aopalliance</artifactId>
            <groupId>aopalliance</groupId>
          </exclusion>
          <exclusion>
            <artifactId>commons-logging</artifactId>
            <groupId>commons-logging</groupId>
          </exclusion>
          <exclusion>
            <artifactId>paranamer</artifactId>
            <groupId>com.thoughtworks.paranamer</groupId>
          </exclusion>
        </exclusions>
      </dependency>
      <dependency>
        <groupId>org.bouncycastle</groupId>
        <artifactId>bcprov-jdk15on</artifactId>
        <version>1.60</version>
      </dependency>
      <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-mapreduce-client-core</artifactId>
        <version>2.6.5</version>
      </dependency>
      <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>${scala.major.version}.${scala.minor.version}.${scala.patch.version}</version>
      </dependency>
      <dependency>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro</artifactId>
        <version>1.7.7</version>
      </dependency>
      <dependency>
        <groupId>org.xerial.snappy</groupId>
        <artifactId>snappy-java</artifactId>
        <version>1.1.2.6</version>
      </dependency>
      <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-math3</artifactId>
        <version>3.4.1</version>
      </dependency>
      <dependency>
        <groupId>commons-lang</groupId>
        <artifactId>commons-lang</artifactId>
        <version>2.6</version>
      </dependency>
      <dependency>
        <groupId>com.google.code.findbugs</groupId>
        <artifactId>jsr305</artifactId>
        <version>3.0.2</version>
      </dependency>
      <dependency>
        <groupId>org.scala-lang.modules</groupId>
        <artifactId>scala-parser-combinators_${scala.major.version}.${scala.minor.version}</artifactId>
        <version>1.0.4</version>
      </dependency>
      <dependency>
        <groupId>org.codehaus.janino</groupId>
        <artifactId>commons-compiler</artifactId>
        <version>3.0.8</version>
      </dependency>
      <dependency>
        <groupId>org.glassfish.hk2.external</groupId>
        <artifactId>javax.inject</artifactId>
        <version>2.5.0-b32</version>
      </dependency>
      <dependency>
        <groupId>org.glassfish.jersey.core</groupId>
        <artifactId>jersey-common</artifactId>
        <version>2.25.1</version>
      </dependency>
      <dependency>
        <groupId>io.netty</groupId>
        <artifactId>netty</artifactId>
        <version>3.9.9.Final</version>
      </dependency>
      <dependency>
        <groupId>javax.activation</groupId>
        <artifactId>activation</artifactId>
        <version>1.1.1</version>
      </dependency>
      <dependency>
        <groupId>org.glassfish.jersey.core</groupId>
        <artifactId>jersey-client</artifactId>
        <version>2.25.1</version>
      </dependency>
      <dependency>
        <groupId>org.codehaus.jackson</groupId>
        <artifactId>jackson-mapper-asl</artifactId>
        <version>1.9.13</version>
      </dependency>
      <dependency>
        <groupId>commons-net</groupId>
        <artifactId>commons-net</artifactId>
        <version>3.1</version>
      </dependency>
      <dependency>
        <groupId>org.codehaus.jackson</groupId>
        <artifactId>jackson-xc</artifactId>
        <version>1.9.13</version>
      </dependency>
      <dependency>
        <groupId>org.codehaus.jackson</groupId>
        <artifactId>jackson-jaxrs</artifactId>
        <version>1.9.13</version>
      </dependency>
      <dependency>
        <groupId>org.codehaus.jackson</groupId>
        <artifactId>jackson-core-asl</artifactId>
        <version>1.9.13</version>
      </dependency>
      <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-reflect</artifactId>
        <version>${scala.major.version}.${scala.minor.version}.${scala.patch.version}</version>
      </dependency>

      <!-- =================================== -->
      <!--        Top level dependencies       -->
      <!-- =================================== -->

      <dependency>
        <groupId>com.typesafe</groupId>
        <artifactId>config</artifactId>
        <version>1.3.0</version>
      </dependency>
      <dependency>
        <groupId>com.oracle</groupId>
        <artifactId>ojdbc6</artifactId>
        <version>11.2.0.3</version>
      </dependency>

      <dependency>
        <groupId>com.databricks</groupId>
        <artifactId>spark-avro_${scala.major.version}.${scala.minor.version}</artifactId>
        <version>4.0.0</version>
      </dependency>

      <!-- =================================== -->
      <!--        Test dependencies            -->
      <!-- =================================== -->
      <dependency>
        <groupId>org.scalatest</groupId>
        <artifactId>scalatest_${scala.major.version}.${scala.minor.version}</artifactId>
        <version>3.0.5</version>
        <scope>test</scope>
      </dependency>
      <dependency>
        <groupId>org.scalatest</groupId>
        <artifactId>scalatest-maven-plugin</artifactId>
        <version>2.0.0</version>
        <scope>test</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

  <dependencies>
    <dependency>
      <groupId>org.scalatest</groupId>
      <artifactId>scalatest_${scala.major.version}.${scala.minor.version}</artifactId>
      <exclusions>
        <exclusion>
          <artifactId>scala-library</artifactId>
          <groupId>org.scala-lang</groupId>
        </exclusion>
        <exclusion>
          <artifactId>scala-xml_${scala.major.version}.${scala.minor.version}</artifactId>
          <groupId>org.scala-lang.modules</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
    </dependency>
  </dependencies>

  <build>
    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>net.alchim31.maven</groupId>
          <artifactId>scala-maven-plugin</artifactId>
          <version>3.2.2</version>
        </plugin>
        <plugin>
          <groupId>org.scalatest</groupId>
          <artifactId>scalatest-maven-plugin</artifactId>
          <version>2.0.0</version>
        </plugin>
      </plugins>
    </pluginManagement>
    <plugins>
      <plugin>
        <groupId>net.alchim31.maven</groupId>
        <artifactId>scala-maven-plugin</artifactId>
        <executions>
          <execution>
            <goals>
              <goal>compile</goal>
              <goal>testCompile</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <scalaCompatVersion>${scala.major.version}.${scala.minor.version}</scalaCompatVersion>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.basepom.maven</groupId>
        <artifactId>duplicate-finder-maven-plugin</artifactId>
        <configuration>
          <ignoredClassPatterns combine.self="append">
            <ignoredClassPattern>org.apache.spark.unused.UnusedStubClass</ignoredClassPattern>
          </ignoredClassPatterns>
          <ignoredResourcePatterns combine.self="append">
            <!-- mercury internal file (should provide link to code that proves its ok to have more than one on the classpath) -->
            <ignoredResource>overrides.conf</ignoredResource>
            <ignoredResource>git.properties</ignoredResource>
          </ignoredResourcePatterns>
        </configuration>
      </plugin>
    </plugins>
  </build>

</project>

Answer 1

您应该在子pom或当前pom中添加a3和spark-core依赖项（如果未按如下所示添加）

spark-sql

看看Difference between dependencies and dependency management

对于未找到的主类 将<dependencies> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_${scala.major.version}.${scala.minor.version}</artifactId> <version>${spark.version}</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_${scala.major.version}.${scala.minor.version}</artifactId> <version>${spark.version}</version> </dependency> </dependencies> 插件作为

添加到子pom

Answer 2

您可以添加Maven装配罐来构建胖罐。添加以下插件并更新您的主类的名称。

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.6</version>
                <executions>
                    <execution>
                        <id>jar-with-dependencies</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                        <configuration>
                            <descriptorRefs>
                                <descriptorRef>jar-with-dependencies</descriptorRef>
                            </descriptorRefs>
                            <archive>
                                <manifest>
                                    <mainClass>***com.abc.mainclass***</mainClass>
                                </manifest>
                            </archive>
                        </configuration>
                    </execution>
                </executions>
            </plugin>