无法解决SBT离线模式下的依赖关系

时间:2018-04-23 18:32:52

标签: scala apache-spark sbt

我的内容与this problem完全相同,但据称0.13.2已解决所述问题,而我仍然在1.1.2中看到它。基本上,即使首先在线运行相同的构建,我也无法离线构建。

$ cat project/build.properties
sbt.version=1.1.2

$ sbt package
...
[success] Total time: 9 s, completed Apr 23, 2018 2:07:36 PM

$ sbt "set offline := true" package
...
[error] unresolved dependency: org.apache.spark#spark-hive_2.11;2.2.1: not found
[error]         at sbt.internal.librarymanagement.IvyActions$.resolveAndRetrieve(IvyActions.scala:334)
[error]         at sbt.internal.librarymanagement.IvyActions$.$anonfun$updateEither$1(IvyActions.scala:208)
[error]         at sbt.internal.librarymanagement.IvySbt$Module.$anonfun$withModule$1(Ivy.scala:243)
[error]         at sbt.internal.librarymanagement.IvySbt.$anonfun$withIvy$1(Ivy.scala:204)
[error]         at sbt.internal.librarymanagement.IvySbt.sbt$internal$librarymanagement$IvySbt$$action$1(Ivy.scala:70)
[error]         at sbt.internal.librarymanagement.IvySbt$$anon$3.call(Ivy.scala:77)
[error]         at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:95)
[error]         at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:80)
[error]         at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:99)
[error]         at xsbt.boot.Using$.withResource(Using.scala:10)
[error]         at xsbt.boot.Using$.apply(Using.scala:9)
[error]         at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:60)
[error]         at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:50)
[error]         at xsbt.boot.Locks$.apply0(Locks.scala:31)
[error]         at xsbt.boot.Locks$.apply(Locks.scala:28)
[error]         at sbt.internal.librarymanagement.IvySbt.withDefaultLogger(Ivy.scala:77)
[error]         at sbt.internal.librarymanagement.IvySbt.withIvy(Ivy.scala:199)
[error]         at sbt.internal.librarymanagement.IvySbt.withIvy(Ivy.scala:196)
[error]         at sbt.internal.librarymanagement.IvySbt$Module.withModule(Ivy.scala:242)
[error]         at sbt.internal.librarymanagement.IvyActions$.updateEither(IvyActions.scala:193)
[error]         at sbt.librarymanagement.ivy.IvyDependencyResolution.update(IvyDependencyResolution.scala:20)
[error]         at sbt.librarymanagement.DependencyResolution.update(DependencyResolution.scala:56)
[error]         at sbt.internal.LibraryManagement$.resolve$1(LibraryManagement.scala:46)
[error]         at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$12(LibraryManagement.scala:99)
[error]         at sbt.util.Tracked$.$anonfun$lastOutput$1(Tracked.scala:68)
[error]         at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$19(LibraryManagement.scala:112)
[error]         at scala.util.control.Exception$Catch.apply(Exception.scala:224)
[error]         at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$11(LibraryManagement.scala:112)
[error]         at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$11$adapted(LibraryManagement.scala:95)
[error]         at sbt.util.Tracked$.$anonfun$inputChanged$1(Tracked.scala:149)
[error]         at sbt.internal.LibraryManagement$.cachedUpdate(LibraryManagement.scala:126)
[error]         at sbt.Classpaths$.$anonfun$updateTask$5(Defaults.scala:2386)
[error]         at scala.Function1.$anonfun$compose$1(Function1.scala:44)
[error]         at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:39)
[error]         at sbt.std.Transform$$anon$4.work(System.scala:66)
[error]         at sbt.Execute.$anonfun$submit$2(Execute.scala:262)
[error]         at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
[error]         at sbt.Execute.work(Execute.scala:271)
[error]         at sbt.Execute.$anonfun$submit$1(Execute.scala:262)
[error]         at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:174)
[error]         at sbt.CompletionService$$anon$2.call(CompletionService.scala:36)
[error]         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[error]         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error]         at java.lang.Thread.run(Thread.java:748)
[error] (update) sbt.librarymanagement.ResolveException: unresolved dependency: org.apache.spark#spark-sql_2.11;2.2.1: not found

这是我的sbt文件:

name := "spark-tools"
version := "1.0"
scalaVersion := "2.11.8"

val scalaBuildVersion = "2.11"
val sparkVersion = "2.2.1"
val hadoopVersion = "2.7.3"

// OK
libraryDependencies += "org.apache.hadoop" % "hadoop-aws" % s"$hadoopVersion"
libraryDependencies += "com.amazonaws" % "aws-java-sdk" % "1.7.4"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.2.1"
libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.4"
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.4"

// Missing
libraryDependencies += "org.apache.spark" % s"spark-core_$scalaBuildVersion" % s"$sparkVersion"
libraryDependencies += "org.apache.spark" % s"spark-sql_$scalaBuildVersion" % s"$sparkVersion"
libraryDependencies += "org.apache.spark" % s"spark-hive_$scalaBuildVersion" % s"$sparkVersion"

带有-2.2.1后缀的三个jar都会导致未解析的依赖关系,但它们都在常春藤缓存中:

$ find ~/.ivy2 -name "spark-core_2.11-2.2.1.jar"
~/.ivy2/cache/org.apache.spark/spark-core_2.11/jars/spark-core_2.11-2.2.1.jar

$ find ~/.ivy2 -name "spark-sql_2.11-2.2.1.jar"
~/.ivy2/cache/org.apache.spark/spark-sql_2.11/jars/spark-sql_2.11-2.2.1.jar

$ find ~/.ivy2 -name "spark-hive_2.11-2.2.1.jar"
~/.ivy2/cache/org.apache.spark/spark-hive_2.11/jars/spark-hive_2.11-2.2.1.jar

不知道为什么这些罐子的处理方式与其他罐子不同。任何想法都赞赏......

0 个答案:

没有答案