Question

， Spark代码编译操作在少数机器上失败，而相同的源代码在少数其他机器上传递。 请检查Centos上的错误（4.10.12-1.el7.elrepo.x86_64） ./build/mvn -X -DskipTests -Dscala.lib.directory = / usr / share / scala -pl core compile

INFO] Building Spark Project Core 2.2.2-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[WARNING] The POM for org.apache.spark:spark-launcher_2.11:jar:2.2.2-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.apache.spark:spark-network-common_2.11:jar:2.2.2-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.apache.spark:spark-network-shuffle_2.11:jar:2.2.2-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.apache.spark:spark-unsafe_2.11:jar:2.2.2-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.apache.spark:spark-tags_2.11:jar:2.2.2-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.apache.spark:spark-tags_2.11:jar:tests:2.2.2-SNAPSHOT is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.804 s
[INFO] Finished at: 2017-12-04T23:18:58-08:00
[INFO] Final Memory: 43M/1963M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project spark-core_2.11: Could not resolve dependencies for project org.apache.spark:spark-core_2.11:jar:2.2.2-SNAPSHOT: The following artifacts could not be resolved: org.apache.spark:spark-launcher_2.11:jar:2.2.2-SNAPSHOT, org.apache.spark:spark-network-common_2.11:jar:2.2.2-SNAPSHOT, org.apache.spark:spark-network-shuffle_2.11:jar:2.2.2-SNAPSHOT, org.apache.spark:spark-unsafe_2.11:jar:2.2.2-SNAPSHOT, org.apache.spark:spark-tags_2.11:jar:2.2.2-SNAPSHOT, org.apache.spark:spark-tags_2.11:jar:tests:2.2.2-SNAPSHOT: Failure to find org.apache.spark:spark-launcher_2.11:jar:2.2.2-SNAPSHOT in http://artifact.eng.stellus.in:8081/artifactory/libs-snapshot was cached in the local repository, resolution will not be reattempted until the update interval of snapshots has elapsed or updates are forced -> [Help 1]

注意：相同的源代码在另一台CentOS机器上传递（3.10.0-514.el7.x86_64） ./build/mvn -DskipTests -Dscala.lib.directory = / usr / share / scala -pl core compile

[INFO] — maven-compiler-plugin:3.7.0:compile (default-compile) @ spark-core_2.11 —
[INFO] Not compiling main sources
[INFO]
[INFO] — scala-maven-plugin:3.2.2:compile (scala-compile-first) @ spark-core_2.11 —
[INFO] Using zinc server for incremental compilation
[info] Compile success at Dec 4, 2017 11:17:34 PM [0.331s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5.663 s
[INFO] Finished at: 2017-12-04T23:17:34-08:00
[INFO] Final Memory: 52M/1297M
[INFO] ------------------------------------------------------------------------

Answer 1

如果构建仅在特定计算机上运行，则意味着您已经缺少对.m2本地存储库的依赖。如果您将删除.m2下的所有目录（在centos机器上），则构建将失败。

您应该添加所有缺少的依赖项。

[WARNING] The POM for org.apache.spark:spark-launcher_2.11:jar:2.2.2-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.apache.spark:spark-network-common_2.11:jar:2.2.2-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.apache.spark:spark-network-shuffle_2.11:jar:2.2.2-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.apache.spark:spark-unsafe_2.11:jar:2.2.2-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.apache.spark:spark-tags_2.11:jar:2.2.2-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.apache.spark:spark-tags_2.11:jar:tests:2.2.2-SNAPSHOT is missing, no dependency information available

Answer 2

以下补丁修复了编译错误

diff --git a/pom.xml b/pom.xml
index cc48ee794e..ef0b01bec8 100644
--- a/pom.xml
+++ b/pom.xml
@@ -2518,6 +2518,13 @@
           <groupId>org.apache.maven.plugins</groupId>
           <artifactId>maven-jar-plugin</artifactId>
           <version>3.1.2</version>
+                 <executions>
+                   <execution>
+                     <goals>
+                       <goal>test-jar</goal>
+                     </goals>
+                   </execution>
+                 </executions>
         </plugin>
         <plugin>
           <groupId>org.apache.maven.plugins</groupId>

请参阅此处stackoverflow.com/questions/4786881 this link in Stack Overflow

另一种可能性：您有一个执行<maven.test.skip>true</maven.test.skip>的Maven配置文件（跳过测试编译和执行）
代替 <skipTests>true</skipTests> （跳过执行，但编译测试）

无法在项目spark-core_2.11上执行目标

2 个答案: