Question

我刚开始在HDP 2.6上使用Spark 2.2，而Iam在尝试进行sbt编译时遇到了问题

错误

[info]更新了文件/home/maria_dev/structuredstreaming/project/build.properties：将sbt.version设置为1.3.0 [info]从/ home / maria_dev / structuredstreaming / project加载项目定义 [info]正在获取以下物品 [info]提取的物品 [错误] lmcoursier.internal.shaded.coursier.error.FetchError $ DownloadingArtifacts：提取工件时出错： [错误] https://repo1.maven.org/maven2/com/squareup/okhttp3/okhttp-urlconnection/3.7.0/okhttp-urlconnection-3.7.0.jar：下载错误：捕获到java.net.UnknownHostException：下载https://repo1.maven.org/maven2/com/squareup/okhttp3/okhttp-urlconnection/3.7.0/okhttp-urlconnection-3.7.0.jar时出现repo1.maven.org（repo1.maven.org）

build.sbt文件如下

buid.sbt

scalaVersion := "2.11.8"

resolvers ++= Seq(
 "Conjars" at "http://conjars.org/repo",
 "Hortonworks Releases" at "http://repo.hortonworks.com/content/groups/public"
)

publishMavenStyle := true

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "2.2.0.2.6.3.0-235",
  "org.apache.spark" %% "spark-sql" % "2.2.0.2.6.3.0-235",
  "org.apache.phoenix" % "phoenix-spark2" % "4.7.0.2.6.3.0-235",
  "org.apache.phoenix" % "phoenix-core" % "4.7.0.2.6.3.0-235",
  "org.apache.kafka" % "kafka-clients" % "0.10.1.2.6.3.0-235",
  "org.apache.spark" %% "spark-streaming" % "2.0.2" % "provided",
  "org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.0.2",
  "org.apache.spark" %% "spark-sql-kafka-0-10" % "2.0.2" % "provided",
  "com.typesafe" % "config" % "1.3.1",
  "com.typesafe.play" %% "play-json" % "2.7.2",
  "com.solarmosaic.client" %% "mail-client" % "0.1.0",
  "org.json4s" %% "json4s-jackson" % "3.2.10",
  "org.apache.logging.log4j" % "log4j-api-scala_2.11" % "11.0",
  "com.databricks" %% "spark-avro" % "3.2.0",
  "org.elasticsearch" %% "elasticsearch-spark-20" % "5.0.0-alpha5",
  "io.spray" %%  "spray-json" % "1.3.3"
)

retrieveManaged := true

fork in run := true

Answer 1

看起来，coursier试图从被阻止的repo1.maven.org中获取依赖项。 Scala-Metals人员进行了解释here。基本上，您必须通过设置如下所示的mirror.properties文件来设置指向公司代理服务器的全局Coursier配置：

central.from=https://repo1.maven.org/maven2
central.to=http://mycorporaterepo.com:8080/nexus/content/groups/public

根据您的操作系统，它将是：

Windows：C：\ Users \\ AppData \ Roaming \ Coursier \ config \ mirror.properties
Linux：〜/ .config / coursier / mirror.properties
MacOS：〜/ Library / Preferences / Coursier / mirror.properties

您可能还需要设置SBT以使用代理来下载依赖项。为此，您将需要编辑以下文件：

~/.sbt/repositories

将其设置为以下内容：

[repositories]
  local
  maven-central: http://mycorporaterepo.com:8080/nexus/content/groups/public

这两种设置的结合应该可以帮助您将SBT指向正确的位置。

Spark 2.2的SBT编译失败

1 个答案: