Question

我写了以下build.sbt文件

lazy val root = (project in file(".")).settings(
    name := "SparkStreaming1",
    organization := "abhi",
    version := "1.0",
    scalaVersion := "2.11.8",
    libraryDependencies ++= Seq(
      "org.apache.spark" %% "spark-streaming" % "1.6.0" % "provided",
      "org.apache.spark" %% "spark-core" % "1.6.0" % "provided",
      "org.apache.spark" %% "spark-streaming-twitter" % "1.6.0" force()
    ),
    mainClass in assembly := Some("com.abhi.TendingHashTags"),
    assemblyMergeStrategy in assembly := {
      case PathList("javax", "servlet", xs @ _*)         => MergeStrategy.first
      case PathList(ps @ _*) if ps.last endsWith ".html" => MergeStrategy.first
      case "application.conf"                            => MergeStrategy.concat
      case "UnusedStubClass.class"                       => MergeStrategy.first
      case "unwanted.txt"                                => MergeStrategy.discard
      case x =>
        val oldStrategy = (assemblyMergeStrategy in assembly).value
        oldStrategy(x)
    }
  )

您可以看到我为UnusedStubClass.class定义了合并策略但仍然在sbt clean assembly时出现错误

[error] 1 error was encountered during merge
java.lang.RuntimeException: deduplicate: different file contents found in the following:
/Users/abhishek.srivastava/.ivy2/cache/org.apache.spark/spark-streaming-twitter_2.11/jars/spark-streaming-twitter_2.11-1.6.0.jar:org/apache/spark/unused/UnusedStubClass.class
/Users/abhishek.srivastava/.ivy2/cache/org.spark-project.spark/unused/jars/unused-1.0.0.jar:org/apache/spark/unused/UnusedStubClass.class
    at sbtassembly.Assembly$.applyStrategies(Assembly.scala:140)
    at sbtassembly.Assembly$.x$1$lzycompute$1(Assembly.scala:25)
    at sbtassembly.Assembly$.x$1$1(Assembly.scala:23)
    at sbtassembly.Assembly$.stratMapping$lzycompute$1(Assembly.scala:23)
    at sbtassembly.Assembly$.stratMapping$1(Assembly.scala:23)
    at sbtassembly.Assembly$.inputs$lzycompute$1(Assembly.scala:67)
    at sbtassembly.Assembly$.inputs$1(Assembly.scala:57)
    at sbtassembly.Assembly$.apply(Assembly.scala:83)
    at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:241)
    at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:238)
    at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
    at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
    at sbt.std.Transform$$anon$4.work(System.scala:63)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
    at sbt.Execute.work(Execute.scala:235)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] /Users/abhishek.srivastava/.ivy2/cache/org.apache.spark/spark-streaming-twitter_2.11/jars/spark-streaming-twitter_2.11-1.6.0.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/abhishek.srivastava/.ivy2/cache/org.spark-project.spark/unused/jars/unused-1.0.0.jar:org/apache/spark/unused/UnusedStubClass.class
[error] Total time: 12 s, completed May 10, 2016 11:01:09 PM

为什么我指定的assemblyMergeStrategy不起作用？

Answer 1

因为您在根目录中有UnusedStubClass.class的策略，而不是org/apache/spark/unused/UnusedStubClass.class。请改用PathList("org", "apache", "spark", "unused", "UnusedStubClass.class")。或者类似PathList(xs @ _*) if xs.last == "UnusedStubClass.class"，如果您想在任何包中处理UnusedStubClass.class。

sbt assemblyMergeStrategy无法正常工作

1 个答案: