找不到参数sparkSession的隐式值

Question

我有一个笔记本，下面的代码引发以下错误：

  

找不到参数sparkSession的隐式值

import org.apache.spark.sql.{SparkSession, Row, DataFrame}
import org.apache.spark.ml.clustering.KMeans

def createBalancedDataframe(df:DataFrame, reductionCount:Int)(implicit sparkSession:SparkSession) = {

    val kMeans = new KMeans().setK(reductionCount).setMaxIter(30)
    val kMeansModel = kMeans.fit(df)

    import sparkSession.implicits._
    kMeansModel.clusterCenters.toList.map(v => (v, 0)).toDF("features", "label")
  }

val balancedNonFraudDF = createBalancedDataframe(nonFraudDF, fraudCount.toInt)

错误：

Name: Compile Error
Message: <console>:82: error: could not find implicit value for parameter sparkSession: org.apache.spark.sql.SparkSession
       val balancedNonFraudDF = createBalancedDataframe(nonFraudDF, fraudCount.toInt)
                                                       ^

StackTrace: 

非常感谢任何人能提供任何帮助，在此先感谢您。

更新：

在我将其更改为

后，感谢Reddy的输入

val balancedNonFraudDF = createBalancedDataframe(nonFraudDF, fraudCount.toInt)(spark)

我收到以下错误：

Name: java.lang.IllegalArgumentException
Message: Field "features" does not exist.
Available fields: cc_num, trans_num, trans_time, category, merchant, amt, merch_lat, merch_long, distance, age, is_fraud
StackTrace: Available fields: cc_num, trans_num, trans_time, category, merchant, amt, merch_lat, merch_long, distance, age, is_fraud
  at org.apache.spark.sql.types.StructType$$anonfun$apply$1.apply(StructType.scala:267)
  at org.apache.spark.sql.types.StructType$$anonfun$apply$1.apply(StructType.scala:267)
  at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
  at scala.collection.AbstractMap.getOrElse(Map.scala:59)
  at org.apache.spark.sql.types.StructType.apply(StructType.scala:266)
  at org.apache.spark.ml.util.SchemaUtils$.checkColumnType(SchemaUtils.scala:40)
  at org.apache.spark.ml.clustering.KMeansParams$class.validateAndTransformSchema(KMeans.scala:93)
  at org.apache.spark.ml.clustering.KMeans.validateAndTransformSchema(KMeans.scala:254)
  at org.apache.spark.ml.clustering.KMeans.transformSchema(KMeans.scala:340)
  at org.apache.spark.ml.PipelineStage.transformSchema(Pipeline.scala:74)
  at org.apache.spark.ml.clustering.KMeans.fit(KMeans.scala:305)
  at createBalancedDataframe(<console>:45)

UPDATE2：

featureDF.printSchema
root
 |-- cc_num: long (nullable = true)
 |-- category: string (nullable = true)
 |-- merchant: string (nullable = true)
 |-- distance: double (nullable = true)
 |-- amt: integer (nullable = true)
 |-- age: integer (nullable = true)
 |-- is_fraud: integer (nullable = true)
 |-- category_indexed: double (nullable = false)
 |-- category_encoded: vector (nullable = true)
 |-- merchant_indexed: double (nullable = false)
 |-- merchant_encoded: vector (nullable = true)
 |-- features: vector (nullable = true)

val fraudDF = featureDF
      .filter($"is_fraud" === 1)
      .withColumnRenamed("is_fraud", "label")
      .select("features", "label")

fraudDF.printSchema
root
 |-- cc_num: long (nullable = true)
 |-- trans_num: string (nullable = true)
 |-- trans_time: string (nullable = true)
 |-- category: string (nullable = true)
 |-- merchant: string (nullable = true)
 |-- amt: integer (nullable = true)
 |-- merch_lat: double (nullable = true)
 |-- merch_long: double (nullable = true)
 |-- distance: double (nullable = true)
 |-- age: integer (nullable = true)
 |-- is_fraud: integer (nullable = true)

为什么该功能消失了？

Answer 1

假设您拥有SparkSession并被命名为spark

您可以通过这种方式明确传递

val balancedNonFraudDF = createBalancedDataframe(nonFraudDF, fraudCount.toInt)(spark)

或在调用环境中创建隐式引用（spark2或任何名称）。示例：

implicit val spark2 = spark
//some calls
// others
val balancedNonFraudDF = createBalancedDataframe(nonFraudDF, fraudCount.toInt)