Question

当我做sbt包或sbt程序集时，我得到的错误如下：

value predictProbabilities不是org.apache.spark.mllib.classification.NaiveBayesModel

的成员

但是同样的代码在我的spark-shell中起作用

/home/user1/spark-1.5.2/bin/spark-shell --packages com.databricks：spark-csv_2.10：1.4.0

我使用spark-1.5.2和scala 2.10.4 sbt 0.13.8

我使用centos 6.5 linux

我确实看到此链接中的predictProbabilities。 https://spark.apache.org/docs/1.5.2/api/scala/index.html#org.apache.spark.mllib.classification.NaiveBayesModel

我不知道是否需要sbt assembly或io.prediction 但我试过了。没什么区别。得到同样的错误。

有没有人遇到过这样的问题。如果是这样，你能让我知道解决方案。

谢谢！

我的代码是

=============================================== ===========

import org.apache.spark.mllib.classification.{NaiveBayes, NaiveBayesModel}

import org.apache.spark.mllib.util.MLUtils

val trainData = MLUtils.loadLibSVMFile(sc, "file:///home/user1/nbtest/trainData.nb.svm")

val testData = MLUtils.loadLibSVMFile(sc, "file:///home/user1/nbtest/testData.nb.svm")

val model = NaiveBayes.train(trainData, lambda = 1.0, modelType = "multinomial")

val predictionAndLabel = testData.map(p => (model.predict(p.features), p.label))

val accuracy = 1.0 * predictionAndLabel.filter(x => x._1 == x._2).count() / testData.count()

val testDataFeatures = testData.map(p => p.features)

val probabilities = model.predictProbabilities (testDataFeatures)

=============================================== ====================

我的/home/user1/nbtest/build.sbt就像这样

=============================================== ==============

name := "testBayes"

version := "1.0" 

scalaVersion := "2.10.4"

scalaSource in Compile := baseDirectory.value / ""

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.2"

libraryDependencies += "org.apache.spark" % "spark-sql_2.10" % "1.5.2"

libraryDependencies += "org.apache.spark"  % "spark-mllib_2.10" % "1.4.0"

libraryDependencies += "com.databricks"  % "spark-csv_2.10" % "1.4.0"

libraryDependencies += "io.prediction" % "core_2.10" % "0.9.6"

libraryDependencies += "io.prediction" % "sdk" % "0.5.0"   

assemblyMergeStrategy in assembly := {
 case PathList("META-INF", xs @ _*) => MergeStrategy.discard
 case x => MergeStrategy.first

=============================================== ==============

我在/ home / user1 / nbtest / project中 assembly.sbt plugins.sbt

assembly.sbt

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0")

plugins.sbt

logLevel := Level.Warn

=============================================== ==============

Answer 1

使用此库代替您正在使用的spark-mllib 1.4.0：

libraryDependencies += "org.apache.spark"  % "spark-mllib_2.10" % "1.5.2"

apache spark predictProbabilities不是成员NaiveBayesModel

1 个答案: