Question

我是scala的新手，并尝试使用Scala version 2.11.12和spark version 2.4.3和JDK 1.8.0_212在集群上运行Spark App。我的build.sbt看起来像这样-

name := "LearningScala"
version := "1.0"
scalaVersion := "2.11.12"
val sparkVersion = "2.4.3"

//resolvers += "Maven Dependencies" at "https://repo1.maven.org/maven2/"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-sql" % sparkVersion
)

如您所见，我也尝试使用resolvers，但没有任何效果。错误日志显示以下是未解决的依赖项-

[error] sbt.librarymanagement.ResolveException: unresolved dependency: org.codehaus.jackson#jackson-core-asl;1.9.13: configuration not found in org.codehaus.jackson#jackson-core-asl;1.9.13: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.parquet#parquet-hadoop;1.10.1 compile
[error] unresolved dependency: org.codehaus.jackson#jackson-mapper-asl;1.9.13: configuration not found in org.codehaus.jackson#jackson-mapper-asl;1.9.13: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.parquet#parquet-hadoop;1.10.1 compile
[error] unresolved dependency: commons-httpclient#commons-httpclient;3.1: configuration not found in commons-httpclient#commons-httpclient;3.1: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.hadoop#hadoop-common;2.6.5 compile
[error] unresolved dependency: commons-collections#commons-collections;3.2.2: configuration not found in commons-collections#commons-collections;3.2.2: 'master(compile)'. Missing configuration: 'compile'. It was required from commons-configuration#commons-configuration;1.6 compile
[error] unresolved dependency: log4j#log4j;1.2.17: configuration not found in log4j#log4j;1.2.17: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.hadoop#hadoop-auth;2.6.5 runtime
[error] unresolved dependency: commons-lang#commons-lang;2.6: configuration not found in commons-lang#commons-lang;2.6: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.orc#orc-mapreduce;1.5.5 compile
[error] unresolved dependency: javax.inject#javax.inject;1: configuration not found in javax.inject#javax.inject;1: 'master(compile)'. Missing configuration: 'compile'. It was required from com.google.inject#guice;3.0 compile
[error] unresolved dependency: org.xerial.snappy#snappy-java;1.1.7.3: configuration not found in org.xerial.snappy#snappy-java;1.1.7.3: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.parquet#parquet-hadoop;1.10.1 compile
[error] unresolved dependency: org.lz4#lz4-java;1.4.0: configuration not found in org.lz4#lz4-java;1.4.0: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.spark#spark-core_2.11;2.4.3 compile
[error] unresolved dependency: com.thoughtworks.paranamer#paranamer;2.8: configuration not found in com.thoughtworks.paranamer#paranamer;2.8: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.spark#spark-core_2.11;2.4.3 runtime
[error] unresolved dependency: javax.ws.rs#javax.ws.rs-api;2.0.1: configuration not found in javax.ws.rs#javax.ws.rs-api;2.0.1: 'master(compile)'. Missing configuration: 'compile'. It was required from org.glassfish.jersey.containers#jersey-container-servlet;2.22.2 compile
[error] unresolved dependency: javax.annotation#javax.annotation-api;1.2: configuration not found in javax.annotation#javax.annotation-api;1.2: 'master(compile)'. Missing configuration: 'compile'. It was required from org.glassfish.jersey.core#jersey-server;2.22.2 compile
[error] unresolved dependency: javax.validation#validation-api;1.1.0.Final: configuration not found in javax.validation#validation-api;1.1.0.Final: 'master(compile)'. Missing configuration: 'compile'. It was required from org.glassfish.jersey.core#jersey-server;2.22.2 compile
[error] unresolved dependency: io.dropwizard.metrics#metrics-json;3.1.5: configuration not found in io.dropwizard.metrics#metrics-json;3.1.5:

有人将2.11和2.4.3一起使用吗？我的Spark Shell显示了这些值，并且可以手动导入这样的包-

$ spark-shell
scala> scala> import org.apache.spark._
import org.apache.spark._

scala> import org.apache.spark.sql._
import org.apache.spark.sql._

那为什么build.sbt无法获取它？

Answer 1

在解析器之前删除评论。另外，尝试以这种方式添加存储库

resolvers ++= Seq(
  DefaultMavenRepository,
  Resolver.bintrayRepo("typesafe", "releases"),
  Resolver.sonatypeRepo("releases"),
  Resolver.mavenLocal
)

Answer 2

这是我尝试解决此问题的方法：我先删除了~/.ivy2目录，然后在保留2.2.0的同时开始提高scalaVersion := 2.11.12的Spark版本。完美运行的Spark最终版本为2.4.3。

尽管我尝试了试错法，但我认为2.4.3版本无法较早下载的某些依赖项是由其先前版本下载的。

我尝试并成功运行的Spark版本列表-

2.2.0
2.2.1
2.3.0
2.3.1
2.4.0
2.4.3

我知道这很奇怪，但对我有用。我认为问题出在已经下载到~/.ivy2目录中的缓存打包文件中。仅删除cache/内的.ivy2将无济于事。

火花2.4.3的build.sbt未解决依赖项

2 个答案: