SBT Assembly插件出错

Question

我写了以下sbt文件

name := "Test"

version := "1.0"

scalaVersion := "2.11.7"

libraryDependencies ++= Seq(
  "org.apache.hadoop" % "hadoop-client" % "2.7.1",
  "org.apache.spark" % "spark-core_2.10" % "1.3.0",
  "org.apache.avro" % "avro" % "1.7.7",
  "org.apache.avro" % "avro-mapred" % "1.7.7"
)

mainClass := Some("com.test.Foo")

我在项目文件夹

中也有以下assembly.sbt文件

resolvers += Resolver.url("bintray-sbt-plugins", url("http://dl.bintray.com/sbt/sbt-plugin-releases"))(Resolver.ivyStylePatterns)
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.0")

当我sbt assembly时，我得到了一个巨大的错误列表

[error] (*:assembly) deduplicate: different file contents found in the following:
[error] /Users/abhishek.srivastava/.ivy2/cache/com.esotericsoftware.kryo/kryo/bundles/kryo-2.21.jar:com/esotericsoftware/minlog/Log$Logger.class
[error] /Users/abhishek.srivastava/.ivy2/cache/com.esotericsoftware.minlog/minlog/jars/minlog-1.2.jar:com/esotericsoftware/minlog/Log$Logger.class
[error] deduplicate: different file contents found in the following:
[error] /Users/abhishek.srivastava/.ivy2/cache/com.esotericsoftware.kryo/kryo/bundles/kryo-2.21.jar:com/esotericsoftware/minlog/Log.class
[error] /Users/abhishek.srivastava/.ivy2/cache/com.esotericsoftware.minlog/minlog/jars/minlog-1.2.jar:com/esotericsoftware/minlog/Log.class
[error] deduplicate: different file contents found in the following:

Answer 1

我能够解决问题。实际上没有必要构建一个胖罐，因为“spark-submit”工具无论如何都会在类路径中包含所有东西。

因此构建jar文件的正确方法是

name := "Test"

version := "1.0"

scalaVersion := "2.11.7"

libraryDependencies ++= Seq(
  "org.apache.hadoop" % "hadoop-client" % "2.7.1" % "provided",
  "org.apache.spark" % "spark-core_2.10" % "1.3.0" % "provided",
  "org.apache.avro" % "avro" % "1.7.7" % "provided",
  "org.apache.avro" % "avro-mapred" % "1.7.7" % "provided"
)

mainClass := Some("com.test.Foo")

Answer 2

1。使用MergeStrategy，请参阅sbt-assembly

2。排除重复的jar，例如：

    lazy val hbaseLibSeq = Seq(
    ("org.apache.hbase" % "hbase" % hbaseVersion).
      excludeAll(
        ExclusionRule(organization = "org.slf4j"),
        ExclusionRule(organization = "org.mortbay.jetty"),
        ExclusionRule(organization = "javax.servlet")),
    ("net.java.dev.jets3t" % "jets3t" % "0.6.1" % "provided").
      excludeAll(ExclusionRule(organization = "javax.servlet"))
  )

3。使用提供的范围

显示依赖树：

• ➜cat~/ .sbt / 0.13 / plugins / plugins.sbt

addSbtPlugin（“net.virtual-void”％“sbt-dependency-graph”％“0.7.5”） addSbtPlugin（“com.eed3si9n”％“sbt-assembly”％“0.11.2”）

• ➜cat~/ .sbt / 0.13 / global.sbt

net.virtualvoid.sbt.graph.Plugin.graphSettings

• ➜sbtdependency-graph