Question

我正在尝试构建一个Scala jar文件以在spark中运行它我正在关注tutorial 当尝试使用sbt作为here构建jar文件时，我面临以下错误

[info] Resolving org.apache.spark#spark-core_2.10.4;1.0.2 ...
[warn]  module not found: org.apache.spark#spark-core_2.10.4;1.0.2
[warn] ==== local: tried
[warn]   /home/hduser/.ivy2/local/org.apache.spark/spark-core_2.10.4/1.0.2/ivys/ivy.xml
[warn] ==== Akka Repository: tried
[warn]   http://repo.akka.io/releases/org/apache/spark/spark-core_2.10.4/1.0.2/spark-core_2.10.4-1.0.2.pom
[warn] ==== public: tried
[warn]   http://repo1.maven.org/maven2/org/apache/spark/spark-core_2.10.4/1.0.2/spark-core_2.10.4-1.0.2.pom
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.apache.spark#spark-core_2.10.4;1.0.2: not found
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[error] {file:/home/prithvi/scala/asd/}default-d57abf/*:update: sbt.ResolveException: unresolved dependency: org.apache.spark#spark-core_2.10.4;1.0.2: not found
[error] Total time: 2 s, completed 13 Aug, 2014 5:24:24 PM

问题是什么以及如何解决。

依赖问题已经解决。谢谢“om-nom-nom”
但出现了新的错误

[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::              FAILED DOWNLOADS            ::
[warn]  :: ^ see resolution messages for details  ^ ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.eclipse.jetty.orbit#javax.transaction;1.1.1.v201105210645!javax.transaction.orbit
[warn]  :: org.eclipse.jetty.orbit#javax.servlet;3.0.0.v201112011016!javax.servlet.orbit
[warn]  :: org.eclipse.jetty.orbit#javax.mail.glassfish;1.4.1.v201005082020!javax.mail.glassfish.orbit
[warn]  :: org.eclipse.jetty.orbit#javax.activation;1.1.0.v201105071233!javax.activation.orbit
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[error] {file:/home/prithvi/scala/asd/}default-c011e4/*:update: sbt.ResolveException: download failed: org.eclipse.jetty.orbit#javax.transaction;1.1.1.v201105210645!javax.transaction.orbit
[error] download failed: org.eclipse.jetty.orbit#javax.servlet;3.0.0.v201112011016!javax.servlet.orbit
[error] download failed: org.eclipse.jetty.orbit#javax.mail.glassfish;1.4.1.v201005082020!javax.mail.glassfish.orbit
[error] download failed: org.eclipse.jetty.orbit#javax.activation;1.1.0.v201105071233!javax.activation.orbit
[error] Total time: 855 s, completed 14 Aug, 2014 12:28:33 PM

Answer 1

您的依赖项定义为

"org.apache.spark" %% "spark-core" % "1.0.2"

%%指示sbt将当前scala版本替换为工件名称。显然，spark was build for the whole family of 2.10 scala，没有针对2.10.1,2.10.2 ......

的特定罐子

所以你要做的就是将它重新定义为：

"org.apache.spark" % "spark-core_2.10" % "1.0.2"

Answer 2

spark-core_2.10.4; 1.0.2表示它是在scala 2.10 vesion之上构建的。所以你必须在你的构建文件中指定这个scalaVersion：=“2.10.4”。请检查您的.sbt文件并进行相应更改。

Answer 3

libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.1.0",
"org.eclipse.jetty.orbit" % "javax.servlet" % "3.0.0.v201112011016",
"org.eclipse.jetty.orbit" % "javax.transaction" % "1.1.1.v201105210645",
"org.eclipse.jetty.orbit" % "javax.mail.glassfish" % "1.4.1.v201005082020"

）

Answer 4

如何更改当前的依赖关系？我的意思是，当您为类似的构建文件键入sbt包时：

name := "Simple Project"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.0"

SBT将开始解析并下载各种依赖项。但是如果你发现它不再存在于maven repo中的依赖关系，该怎么办？你在哪里可以改变它尝试的dpencies。

@OP：问题是你的SBT已经过时了。如果您使用apt下载它，您也可以使用apt来删除它。在任何情况下，下载最新的.tgz（而不是.deb）并简单地解压缩，之后将/ sbt / bin /文件夹添加到.bashrc我注意到旧的SBT（.deb和apt-）获取版本）使用较旧的scala版本。您需要手动添加或更改旧SBT尝试查找的依赖项，或者只需更改为最新（不是sooo）SBT。

Answer 5

我有同样的问题。看起来有些错误出现在不同的版本/编译/等等中。

对我来说，以下build.sbt正常工作

name := "My Project"  
version := "1.0"  
scalaVersion := "2.11.8"  
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.3.2"

希望有帮助

尝试创建jar时无法解决依赖关系错误

5 个答案: