Question

我正在为我使用过apache.commons.cli.DefaultParser的spark作业应用程序获取参数，当我运行spark Submit命令时出现问题，但是在IntelliJ上运行spark应用程序时运行得很好

build.sbt

name := "hp"

version := "0.1"

scalaVersion := "2.11.12"

resolvers += "Job Server Bintray" at "https://dl.bintray.com/spark-jobserver/maven"

assemblyJarName in assembly := "myapp.jar"

mainClass in assembly := Some("com.myaapp.jobs.App")
//assemblyShadeRules in assembly ++= Seq(
//  ShadeRule.rename("com.apache.commons.**" -> "shadedcommons.@1")
//    .inLibrary("com.apache" % "commons" % "1.2")
//    .inProject
//)
libraryDependencies ++= Seq(
  "com.fasterxml.jackson.core" % "jackson-core" % "2.1.1",
  "com.fasterxml.jackson.core" % "jackson-annotations" % "2.1.1",
  "com.fasterxml.jackson.core" % "jackson-databind" % "2.1.1",
  "com.fasterxml.jackson.dataformat" % "jackson-dataformat-yaml" % "2.1.1",
  "com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.9.7",
  "commons-cli" % "commons-cli" % "1.2",
  "org.apache.spark" %% "spark-sql" % "2.3.0"
)
libraryDependencies += "org.apache.spark" %% "spark-mllib" % "2.3.0"

assemblyMergeStrategy in assembly := {
  case PathList("META-INF", xs @ _*) => MergeStrategy.discard
  case x => MergeStrategy.first
}

我将参数传递给我的spark-submit，如下所示

spark-submit --master local --driver-memory 2g --executor-memory  2g  --class com.myapp.jobs.App /home/leroy/Documents/projects/myapp/target/scala-2.11/myapp.jar -c /home/leroy/Documents/projects/myapp/src/main/resources/configuration.json

执行火花提交时，出现以下错误

2019-01-12 00:04:19 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.IllegalAccessError: tried to access method org.apache.commons.cli.Options.getOptionGroups()Ljava/util/Collection; from class org.apache.commons.cli.DefaultParser

另外，当我运行 sbt编译时，也会出现以下错误。

[info] Compiling 22 Scala sources and 1 Java source to /home/leroy/Documents/projects/myapp/target/scala-2.11/classes ...
[error] /home/leroy/Documents/projects/hp-app/src/main/scala/com/myapp/jobs/config/ConfigurationFactory.scala:9:8: object DefaultParser is not a member of package org.apache.commons.cli
[error] import org.apache.commons.cli.{DefaultParser, Options}
[error]        ^
[error] /home/leroy/Documents/projects/hp-app/src/main/scala/com/myapp/jobs/config/ConfigurationFactory.scala:18:31: not found: type DefaultParser
[error]   private val cliParser = new DefaultParser

我什至尝试使用阴影插件，以在上面的 build.sbt 文件中进行注释。

我尝试更改代码中的解析器以使用CommandLineParser，但已弃用它。

需要了解在版本冲突方面我是否在构建SBT中缺少任何内容。

运行spark Submit命令时，apache commons cli解析器库存在依赖性问题

0 个答案: