Question

当我尝试使用Main运行sbt类时，我收到此错误。我错过了什么？

Error:scalac: missing or invalid dependency detected while loading class file 'LDA.class'.
Could not access type Logging in package org.apache.spark,
because it (or its dependencies) are missing. Check your build definition for missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'LDA.class' was compiled against an incompatible version of org.apache.spark.

我的build.sbt看起来像这样：

"org.apache.spark"          %% "spark-core"       %   "2.0.1"      % Provided,
"org.apache.spark"          % "spark-mllib_2.11"  % "1.3.0"

Answer 1

您正在尝试在新的Spark Core上运行旧的sparkMllib。即你的mllib和spark核心版本完全不同。

尝试使用：

"org.apache.spark"          %% "spark-core_2.11"       %   "2.0.1",
"org.apache.spark"          %% "spark-mllib_2.11"  % "2.0.1"

Thant可能会解决您的问题！

针对不兼容版本编译的Spark MLLib异常LDA.class

1 个答案: