Question

我现在已经在这个问题上敲了一会儿我正式陷入困境。我正在尝试编译一个包含在Azure Databricks上运行的简单scala / spark作业的jar，包括对CosmosDB的依赖。导入azure-cosmosdb-spark会在编译期间引入冲突的交叉版本错误，我认为这是一些传递依赖的结果。我已经尝试了许多不同的spark和scala版本，但没有帮助，错误信息也没有说明多少。

我重现错误的最小例子只是build.sbt，目前没有编译任何scala类。我build.sbt的一个示例如下：

name := "ranchero"

version := "0.0.1"

scalaVersion := "2.11.8"
val sparkVersion = "2.2.0"

// additional libraries
libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion % "provided",
  "org.apache.spark" %% "spark-sql" %  sparkVersion % "provided",
  "org.apache.spark" %% "spark-mllib" % sparkVersion % "provided",
  "joda-time" % "joda-time" % "2.9.9",
  "org.scalatest" %% "scalatest" % "3.0.0" % "test",
  "com.microsoft.azure" % "azure-cosmosdb-spark_2.2.0_2.11" % "1.1.0"
)


resolvers ++= Seq(
  "apache-snapshots" at "http://repository.apache.org/snapshots/",
  "Maven central" at "http://repo1.maven.org/maven2/",
 )

如果我注释掉cosmosdb依赖项，那么编译就好了。添加了dep，我得到了错误的味道：

[error] Modules were resolved with conflicting cross-version suffixes in {file:/home/*******/development/ranchero/}ranchero:
[error]    org.apache.spark:spark-launcher _2.10, _2.11
[error]    org.json4s:json4s-ast _2.10, _2.11
[error]    org.apache.spark:spark-network-shuffle _2.10, _2.11
[error]    com.twitter:chill _2.10, _2.11
[error]    org.json4s:json4s-jackson _2.10, _2.11
[error]    com.fasterxml.jackson.module:jackson-module-scala _2.10, _2.11
[error]    org.json4s:json4s-core _2.10, _2.11
[error]    org.apache.spark:spark-unsafe _2.10, _2.11
[error]    org.apache.spark:spark-core _2.10, _2.11
[error]    org.apache.spark:spark-network-common _2.10, _2.11
[error] java.lang.RuntimeException: Conflicting cross-version suffixes in: org.apache.spark:spark-launcher, org.json4s:json4s-ast, org.apache.spark:spark-network-shuffle, com.twitter:chill, org.json4s:json4s-jackson, com.fasterxml.jackson.module:jackson-module-scala, org.json4s:json4s-core, org.apache.spark:spark-unsafe, org.apache.spark:spark-core, org.apache.spark:spark-network-common

，遗憾的是，它并没有提供太多帮助。有关如何纠正此问题的任何建议？

Answer 1

让sbt处理所有依赖项的scala版本。尝试用以下方法替换azure-cosmosdb依赖项：

"com.microsoft.azure" %% "azure-cosmosdb-spark_2.2.0" % "1.1.0"

double百分比将告诉sbt处理依赖项的scala版本标记。至少sbt shell以更正的build.sbt开始。

Answer 2

虽然我同意通过%%处理Scala版本，但在我看来这还不够。我终于设法通过手动从CosmosDB库中排除有问题的依赖项来使其正常工作

"com.microsoft.azure" %% "azure-cosmosdb-spark_2.2.0" % "1.1.1"
   exclude ("org.apache.spark", "spark-launcher_2.10")
   exclude ("org.json4s", "json4s-ast_2.10")
   exclude ("org.apache.spark", "spark-network-shuffle_2.10")
   exclude ("com.twitter", "chill_2.10")
   exclude ("org.json4s", "json4s-jackson_2.10")
   exclude ("com.fasterxml.jackson.module", "jackson-module-scala_2.10")
   exclude ("org.json4s", "json4s-core_2.10")
   exclude ("org.apache.spark", "spark-unsafe_2.10")
   exclude ("org.apache.spark", "spark-core_2.10")
   exclude ("org.apache.spark", "spark-network-common_2.10")

这假定您在项目中使用Scala 2.11。我没有解释为什么这样做是必要的。也许由于某种原因，Maven上的2.11 CosmosDB库链接到2.10依赖项...

跨版本与Spark和Azure-Cosmosdb冲突

2 个答案: