Question

我正在尝试使用Scala在Apache Spark上运行K-means。当我使用Spark网站https://spark.apache.org/docs/2.3.0/ml-clustering.html上的示例时一切都很好，但是当我尝试使用cvs文件时，我遇到了这个问题

scala> val censocsv = spark.read.format("csv").option("sep",",").option("inferSchema","true").option("header", "true").load("censodiscapacidad.csv")
2018-10-01 21:58:31 WARN  SizeEstimator:66 - Failed to check whether UseCompressedOops is set; assuming yes
2018-10-01 21:58:49 WARN  ObjectStore:568 - Failed to get database global_temp, returning NoSuchObjectException
censocsv: org.apache.spark.sql.DataFrame = [ANIO: int, DELEGACION: double ... 123 more fields]

scala> val kmeans = new KMeans().setK(2).setSeed(1L)
kmeans: org.apache.spark.ml.clustering.KMeans = kmeans_860c02e56190

scala> val model = kmeans.fit(censocsv)
java.lang.IllegalArgumentException: Field "features" does not exist.
  at org.apache.spark.sql.types.StructType$$anonfun$apply$1.apply(StructType.scala:267)
  at org.apache.spark.sql.types.StructType$$anonfun$apply$1.apply(StructType.scala:267)
  at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
  at scala.collection.AbstractMap.getOrElse(Map.scala:59)
  at org.apache.spark.sql.types.StructType.apply(StructType.scala:266)
  at org.apache.spark.ml.util.SchemaUtils$.checkColumnType(SchemaUtils.scala:40)
  at org.apache.spark.ml.clustering.KMeansParams$class.validateAndTransformSchema(KMeans.scala:93)
  at org.apache.spark.ml.clustering.KMeans.validateAndTransformSchema(KMeans.scala:254)
  at org.apache.spark.ml.clustering.KMeans.transformSchema(KMeans.scala:340)
  at org.apache.spark.ml.PipelineStage.transformSchema(Pipeline.scala:74)
  at org.apache.spark.ml.clustering.KMeans.fit(KMeans.scala:305)
  ... 51 elided

scala> val predictions = model.transform(censocsv)
<console>:31: error: not found: value model
       val predictions = model.transform(censocsv)
                         ^

scala>

Answer 1

这看起来像Field "features" does not exist. SparkML的副本
您需要在DataFrame中添加一个包含要素列的Vector。

在Scala中使用CSV文件

1 个答案: