Question

包含路径中存在需要进行重复数据删除的冲突jline jar文件。我试图这样做：

我正在构建一个组合的spark和kafka fat jar，并且jline jar文件是重复的。

这是build.sbt文件

import sbt._
import sbt.Keys._
import java.io.File
import AssemblyKeys._

name := "kafkascala"

version      := "0.1.0-SNAPSHOT"

scalaVersion := "2.10.4"

libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.10" % "0.9.1",
 "org.apache.spark" % "spark-examples_2.10" % "0.9.1",
 "org.apache.spark" % "spark-tools_2.10" % "0.9.1",
 "org.apache.kafka" % "kafka_2.10" % "0.8.1.1"   intransitive()
withSources
)

resolvers += "Apache repo" at "https://repository.apache.org/content/repositories/releases"

assemblySettings

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) => {
    case PathList("maven","jline","jline","pom.properties") => MergeStrategy.discard
    case x => old(x)
  }
}

以下是运行 sbt assembly

的错误输出

java.lang.RuntimeException: deduplicate: different file contents found in the following:
C:\Users\S80035683\.ivy2\cache\jline\jline\jars\jline-0.9.94.jar:META-INF/maven/jline/jline/pom.properties
C:\Users\S80035683\.ivy2\cache\org.jruby\jruby-complete\jars\jruby-complete-1.6.5.jar:META-INF/maven/jline/jline/pom.properties
        at sbtassembly.Plugin$Assembly$.sbtassembly$Plugin$Assembly$$applyStrategy$1(Plugin.scala:253)
        at sbtassembly.Plugin$Assembly$$anonfun$15.apply(Plugin.scala:270)
        at sbtassembly.Plugin$Assembly$$anonfun$15.apply(Plugin.scala:267)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
        at sbtassembly.Plugin$Assembly$.applyStrategies(Plugin.scala:272)
        at sbtassembly.Plugin$Assembly$.x$4$lzycompute$1(Plugin.scala:172)
        at sbtassembly.Plugin$Assembly$.x$4$1(Plugin.scala:170)
        at sbtassembly.Plugin$Assembly$.stratMapping$lzycompute$1(Plugin.scala:170)
        at sbtassembly.Plugin$Assembly$.stratMapping$1(Plugin.scala:170)
        at sbtassembly.Plugin$Assembly$.inputs$lzycompute$1(Plugin.scala:214)
        at sbtassembly.Plugin$Assembly$.inputs$1(Plugin.scala:204)
        at sbtassembly.Plugin$Assembly$.apply(Plugin.scala:230)
        at sbtassembly.Plugin$Assembly$$anonfun$assemblyTask$1.apply(Plugin.scala:373)
        at sbtassembly.Plugin$Assembly$$anonfun$assemblyTask$1.apply(Plugin.scala:370)
        at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
        at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:42)
        at sbt.std.Transform$$anon$4.work(System.scala:64)
        at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
        at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
        at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
        at sbt.Execute.work(Execute.scala:244)
        at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
        at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
        at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:160)
        at sbt.CompletionService$$anon$2.call(CompletionService.scala:30)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] C:\Users\S80035683\.ivy2\cache\jline\jline\jars\jline-0.9.94.jar:META-INF/maven/jline/jline/pom.properties
[error] C:\Users\S80035683\.ivy2\cache\org.jruby\jruby-complete\jars\jruby-complete-1.6.5.jar:META-INF/maven/jline/jline/pom.properties

以下是试图解决此问题的build.sbt部分：

assemblySettings

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) => {
    case PathList("maven","jline","jline","pom.properties") => MergeStrategy.discard
    case x => old(x)
  }
}

Answer 1

您的PathList错了。您在"META-INF"前面遗漏了"maven"。这样的事情应该有效：

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) => {
    case PathList("META-INF", "maven","jline","jline", "pom.properties" ) => MergeStrategy.discard
    case PathList("META-INF", "maven","jline","jline", "pom.xml" ) => MergeStrategy.discard
    case x => old(x)
  } 
}

或更简洁（在一行中排除pom.properties和pom.xml）：

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) => {
    case PathList("META-INF", "maven","jline","jline", ps) if ps.startsWith("pom") => MergeStrategy.discard
    case x => old(x)
  } 
}

Answer 2

我得出结论，sbt中的defaultMergeStrategy不必要地导致重复数据删除错误。根据sbt汇编作者的说法，决定失败而不是先选择是因为他们认为用户应该明确决定，但是大多数用户甚至不知道错误意味着什么并且花费他们的年龄来确定改变{{1到deduplicate只是让事情发挥作用。我认为使构建失败是一个错误，以下是first的外观：

defaultMergeStrategy

build.sbt中的重复数据删除

2 个答案: