Question

我正在尝试构建一个＆＃34; hello world＆＃34; -esque app，它使用Spark流来传输来自Kafka代理的数据（这个工作），过滤/处理这些数据，并将其推送到一个（本地）Web浏览器，使用Scalatra Web框架及其支持的Atmosphere Web套接字功能。 Kafka / Spark块独立工作，Scalatra / Atmosphere块也可独立工作。当我尝试将两半放在一起时，我遇到了与库相关的问题。

真正的问题：我如何选择Spark与Scalatra一起玩的库版本？

裸骨Scalatra / Atmosphere应用可以正常工作如下：

organization := "com.example"
name := "example app"
version := "0.1.0"
scalaVersion := "2.12.2"

val ScalatraVersion = "2.5.+"

libraryDependencies ++= Seq(
  "org.json4s"                  %% "json4s-jackson"      % "3.5.2",
  "org.scalatra"                %% "scalatra"            % ScalatraVersion,
  "org.scalatra"                %% "scalatra-scalate"    % ScalatraVersion,
  "org.scalatra"                %% "scalatra-specs2"     % ScalatraVersion    % "test",
  "org.scalatra"                %% "scalatra-atmosphere" % ScalatraVersion,
  "org.eclipse.jetty"           %  "jetty-webapp"        % "9.4.6.v20170531"  % "provided",
  "javax.servlet"               %  "javax.servlet-api"   % "3.1.0"            % "provided"
)

enablePlugins(JettyPlugin)

但是如果我为Spark和Spark流添加新的依赖项，并将Scala版本降低到2.11（Spark-Kafka流式传输所需）：

organization := "com.example"
name := "example app"
version := "0.1.0"
scalaVersion := "2.11.8"

val ScalatraVersion = "2.5.+"
val SparkVersion = "2.2.0"

libraryDependencies ++= Seq(
  "org.json4s"                  %% "json4s-jackson"      % "3.5.2",
  "org.scalatra"                %% "scalatra"            % ScalatraVersion,
  "org.scalatra"                %% "scalatra-scalate"    % ScalatraVersion,
  "org.scalatra"                %% "scalatra-specs2"     % ScalatraVersion    % "test",
  "org.scalatra"                %% "scalatra-atmosphere" % ScalatraVersion,
  "org.eclipse.jetty"           %  "jetty-webapp"        % "9.4.6.v20170531"  % "provided",
  "javax.servlet"               %  "javax.servlet-api"   % "3.1.0"            % "provided"
)

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % SparkVersion,
  "org.apache.spark" %% "spark-streaming" % SparkVersion,
  "org.apache.spark" %% "spark-streaming-kafka-0-8" % SparkVersion
)

enablePlugins(JettyPlugin)

代码编译，但我得到了SBT的驱逐警告：

[warn] There may be incompatibilities among your library dependencies.
[warn] Here are some of the libraries that were evicted:
[warn]  * org.json4s:json4s-jackson_2.11:3.2.11 -> 3.5.3
[warn] Run 'evicted' to see detailed eviction warnings

最后，当Jetty尝试运行Web服务器时，它会因此错误而失败：

WARN:oejuc.AbstractLifeCycle:main: FAILED org.eclipse.jetty.annotations.ServletContainerInitializersStarter@53fb3dab: java.lang.NoClassDefFoundError: com/sun/jersey/spi/inject/InjectableProvider
java.lang.NoClassDefFoundError: com/sun/jersey/spi/inject/InjectableProvider

我如何找到底线？我是Scala世界的新手，依赖的复杂性让我大吃一惊。

Answer 1

删除驱逐警告的一种方法是使用dependencyOverrides

添加具有所需版本的库依赖项

尝试在SBT文件中添加以下内容并重新构建应用程序

dependencyOverrides += "org.json4s" % "json4s-jackson_2.11" % "3.5.3"

检查SBT文档here

如何解决SBT的库依赖警告问题？

1 个答案: