如何构建未发布的Spark RC版本并在sbt项目中使用它？

时间：2017-05-28 14:27:56

标签： maven apache-spark sbt

我正在Spark v2.2.0-rc1上构建一个sbt应用程序。因为，这个版本的Spark在Maven Repository上不可用，所以我从Spark的GitHub存储库下载它并尝试在本地发布它，因此，我可以使用Spark v2.2.0-rc1编译我的应用程序。但是，在执行sbt publishLocal时，我遇到了异常：

$ sbt publishLocal

...
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.postgresql#postgresql;9.4.1207.jre7: several problems occurred while resolving dependency: org.postgresql#postgresql;9.4.1207.jre7 {test=[default(compile)]}:
[warn]  org.postgresql#postgresql;9.4.1207.jre7!postgresql.pom(pom.original) origin location must be absolute: file:/home/user/.m2/repository/org/postgresql/postgresql/9.4.1207.jre7/postgresql-9.4.1207.jre7.pom
[warn]  org.postgresql#postgresql;9.4.1207.jre7!postgresql.pom(pom.original) origin location must be absolute: file:/home/user/.m2/repository/org/postgresql/postgresql/9.4.1207.jre7/postgresql-9.4.1207.jre7.pom
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[info] :: delivering :: org.apache.spark#spark-streaming-flume-assembly_2.11;2.2.0 :: 2.2.0 :: release :: Sun May 28 14:05:05 GMT 2017
[warn] 
[warn]  Note: Unresolved dependencies path:
[warn]      org.postgresql:postgresql:9.4.1207.jre7 ((com.typesafe.sbt.pom.MavenHelper) MavenHelper.scala#L76)
[warn]        +- org.apache.spark:spark-sql_2.11:2.2.0
[info]  delivering ivy file to /home/user/Softwares/spark-2.2.0-rc1/external/flume-assembly/target/scala-2.11/ivy-2.2.0.xml

...

[info]  published ivy to /home/user/.ivy2/local/org.apache.spark/spark-catalyst_2.11/2.2.0/ivys/ivy.xml
sbt.ResolveException: unresolved dependency: org.postgresql#postgresql;9.4.1207.jre7: several problems occurred while resolving dependency: org.postgresql#postgresql;9.4.1207.jre7 {test=[default(compile)]}:
    org.postgresql#postgresql;9.4.1207.jre7!postgresql.pom(pom.original) origin location must be absolute: file:/home/user/.m2/repository/org/postgresql/postgresql/9.4.1207.jre7/postgresql-9.4.1207.jre7.pom
    org.postgresql#postgresql;9.4.1207.jre7!postgresql.pom(pom.original) origin location must be absolute: file:/home/user/.m2/repository/org/postgresql/postgresql/9.4.1207.jre7/postgresql-9.4.1207.jre7.pom

    at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:313)
    at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:191)
    at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:168)
    at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:156)
    at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:156)
    at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:133)
    at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:57)
    at sbt.IvySbt$$anon$4.call(Ivy.scala:65)
    at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
    at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
    at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
    at xsbt.boot.Using$.withResource(Using.scala:10)
    at xsbt.boot.Using$.apply(Using.scala:9)
    at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
    at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
    at xsbt.boot.Locks$.apply0(Locks.scala:31)
    at xsbt.boot.Locks$.apply(Locks.scala:28)
    at sbt.IvySbt.withDefaultLogger(Ivy.scala:65)
    at sbt.IvySbt.withIvy(Ivy.scala:128)
    at sbt.IvySbt.withIvy(Ivy.scala:125)
    at sbt.IvySbt$Module.withModule(Ivy.scala:156)
    at sbt.IvyActions$.updateEither(IvyActions.scala:168)
    at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1481)
    at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1477)
    at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$121.apply(Defaults.scala:1512)
    at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$121.apply(Defaults.scala:1510)
    at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:37)
    at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1515)
    at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1509)
    at sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:60)
    at sbt.Classpaths$.cachedUpdate(Defaults.scala:1532)
    at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1459)
    at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1411)
    at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
    at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
    at sbt.std.Transform$$anon$4.work(System.scala:63)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
    at sbt.Execute.work(Execute.scala:237)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
[error] (sql/*:update) sbt.ResolveException: unresolved dependency: org.postgresql#postgresql;9.4.1207.jre7: several problems occurred while resolving dependency: org.postgresql#postgresql;9.4.1207.jre7 {test=[default(compile)]}:
[error]     org.postgresql#postgresql;9.4.1207.jre7!postgresql.pom(pom.original) origin location must be absolute: file:/home/user/.m2/repository/org/postgresql/postgresql/9.4.1207.jre7/postgresql-9.4.1207.jre7.pom
[error]     org.postgresql#postgresql;9.4.1207.jre7!postgresql.pom(pom.original) origin location must be absolute: file:/home/user/.m2/repository/org/postgresql/postgresql/9.4.1207.jre7/postgresql-9.4.1207.jre7.pom
[error] Total time: 240 s, completed May 28, 2017 2:08:01 PM

然而，当我运行mvn install -DskipTests=true时，它运行得很好，但sbt publishLocal给出了异常。因为，我必须使用sbt构建我的应用程序，我希望spark v2.2.0位于.ivy2文件夹而不是.m2文件夹中。

任何人都可以帮我解决这个问题吗？

1 个答案:

答案 0 :(得分：2)

使用Maven构建Spark（官方方式）如下：

./build/mvn -Phadoop-2.7,yarn,mesos,hive,hive-thriftserver -DskipTests clean install

一旦你在Maven repo中获得了罐子，就在build.sbt中定义一个解析器来使用Maven的本地存储库。

resolvers += Resolver.mavenLocal

请参阅sbt官方文档中的Resolvers。

这并不能解决您在使用PostgreSQL JDBC驱动程序时遇到的问题，但我相信这一点不太重要。

相关问题
最新问题