Question

我正在尝试使用火花创建一个胖子。问题在于，胖子罐子的包装过程似乎从未结束。它只是卡了很长时间。它只是说包装/ abc.jar，似乎从来没有完成以下是我的程序集设置和依赖项

lazy val assemblySettings = Seq(
  assemblyJarName in assembly := name.value + ".jar",
  assemblyOption in assembly := (assemblyOption in assembly).value.copy(cacheUnzip = false),
  assemblyOption in assembly := (assemblyOption in assembly).value.copy(cacheOutput = false),
  assemblyMergeStrategy in assembly := {
    case PathList("META-INF", xs @ _*) => MergeStrategy.discard
    case _                             => MergeStrategy.first
  }
)

这些是我的依赖项

  val sparkSql = "org.apache.spark" %% "spark-sql" % "2.3.0"
  val sparkCore = "org.apache.spark" %% "spark-core" % "2.3.0"
  val sparkRedShift = "com.databricks" %% "spark-redshift" % "0.5.0"
  val sparkHive = "org.apache.spark" %% "spark-hive" % "2.3.0"

  val log4j =  "log4j" % "log4j" % "1.2.17"
  val liftJson = "net.liftweb" %% "lift-json" % "3.3.0-M2"

  val scalaTest = "org.scalatest" %% "scalatest" % "3.0.4" % Test

  val awsSdk = "com.amazonaws" % "aws-java-sdk" % "1.11.289"
  val awsHadoop = "org.apache.hadoop" % "hadoop-aws" % "3.1.0"

  val testDependencies = Seq(scalaTest)
  val awsDependencies = Seq(awsSdk,awsHadoop)
  val sparkDependencies = Seq(sparkCore,sparkHive,sparkSql,sparkRedShift)


  lazy val pipelineCoreDependencies = Seq(liftJson,log4j) ++ testDependencies ++ sparkDependencies ++ awsDependencies

火花-Sbt组装未完成

0 个答案: