Question

我只是想开发一个K-Means算法的简单例子，但是我在加载和清理数据时遇到了很多问题。

这是我的代码：

import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.clustering.KMeans
import org.apache.spark.sql.functions._


val crimeRDD = sc.textFile("/home/borja/spark/pruebas/AlgoritmoClusterizacion/filter_data1.csv")

val header = crimeRDD.first

val data = crimeRDD.filter (justData => justData != header)

//Spark doesn't allow more than 22 element
case class crimeReport (Record_ID: Int, Agency_Name: String, City: String, State: String, Year: Int, Month: String, Crime_Type: String, Crime_Solved: String, Victim_Sex: String, Victim_Age: Int, Victim_Race: String, Perpetrator_Sex: String, Perpetrator_Age: Int, Perpetrator_Race: String, Relationship: String, Victim_Count: Int)

val data_split = data.map(line => line.split(","))

val allData = data_split.map(p => crimeReport(p(0).trim.toInt, p(1).trim.toString, p(2).trim.toString, p(3).trim.toString, p(4).trim.toInt, p(5).trim.toString, p(6).trim.toString, p(7).trim.toString, p(8).trim.toString, p(9).trim.toInt, p(10).trim.toString, p(11).trim.toString, p(12).trim.toInt, p(13).trim.toString,p(14).trim.toString, p(15).trim.toInt))

val allDF = allData.toDF()
allDF.printSchema
//allDF.show(100)

val rowsRDD = allDF.rdd.map(r => (r.getInt(0),r.getString(1),r.getString(2), r.getString(3),r.getInt(4), r.getString(5), r.getString(6), r.getString(7),r.getString(8), r.getInt(9), r.getString(10), r.getString(11),r.getInt(12), r.getString(13),r.getString(14), r.getInt(15)))

rowsRDD.cache()

val features_vector = allDF.rdd.map(r => Vectors.dense(r.getInt(0)))

features_vector.cache()

val KMeansModel = KMeans.train(features_vector,2,40)

但是我收到了这个错误

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 43.0 failed 1 times, most recent failure: Lost task 0.0 in stage 43.0 (TID 57, localhost, executor driver): java.lang.NumberFormatException: For input string: "Jersey"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:272)

我不明白，因为我正在使用 .trim 功能清理白色和空格，是吗？关键是抛出异常是因为当我执行.toInt时有一些String值，对吧？那么如何过滤它们，因为有65k行。

这里有一小组数据：

Record ID,Agency Name,City,State,Year,Month,Crime Type,Crime Solved,Victim Sex,Victim Age,Victim Race,Perpetrator Sex,Perpetrator Age,Perpetrator Race,Relationship,Victim Count
1,Anchorage,Anchorage,Alaska,1980,January,Murder or Manslaughter,Yes,Male,14,Native American/Alaska Native,Male,15,Native American/Alaska Native,Acquaintance,0
13504,Atlantic City,Atlantic,Jersey,1980,January,Murder or Manslaughter,Yes,Male,40,Black,Female,50,Black,Acquaintance,0
13505,Atlantic City,Atlantic,Jersey,1980,January,Murder or Manslaughter,No,Male,23,Black,Unknown,0,Unknown,Unknown,0
13506,Atlantic City,Atlantic,Jersey,1980,January,Murder or Manslaughter,No,Male,52,White,Unknown,0,Unknown,Unknown,0
13507,Atlantic City,Atlantic,Jersey,1980,March,Murder or Manslaughter,Yes,Male,35,Black,Male,23,Black,Unknown,0
13508,Atlantic City,Atlantic,Jersey,1980,March,Murder or Manslaughter,No,Male,25,Black,Unknown,0,Unknown,Unknown,0
13647,Jersey City,Hudson,Jersey,1980,October,Murder or Manslaughter,No,Female,50,White,Unknown,0,Unknown,Unknown,2
13648,Jersey City,Hudson,Jersey,1980,March,Murder or Manslaughter,Yes,Female,60,White,Male,36,White,Father,1
13649,Jersey City,Hudson,Jersey,1980,June,Murder or Manslaughter,Yes,Female,52,Black,Male,26,Black,Unknown,1
13650,Jersey City,Hudson,Jersey,1980,October,Murder or Manslaughter,No,Male,2,White,Unknown,0,Unknown,Unknown,2
13651,Jersey City,Hudson,Jersey,1980,January,Murder or Manslaughter,Yes,Female,68,Black,Male,0,Black,Unknown,0
13652,Jersey City,Hudson,Jersey,1980,January,Murder or Manslaughter,Yes,Female,22,Black,Male,23,Black,Unknown,0
13653,Jersey City,Hudson,Jersey,1980,January,Murder or Manslaughter,Yes,Female,16,White,Male,33,White,Acquaintance,0
13654,Jersey City,Hudson,Jersey,1980,January,Murder or Manslaughter,Yes,Male,34,White,Male,18,White,Acquaintance,0
13655,Jersey City,Hudson,Jersey,1980,February,Murder or Manslaughter,No,Male,29,Black,Unknown,0,Unknown,Unknown,0
13656,Jersey City,Hudson,Jersey,1980,February,Murder or Manslaughter,No,Male,42,White,Unknown,0,Unknown,Unknown,0

Answer 1

是的，当object无法转换为integer值时，会抛出错误消息。为什么不试试Option或Try
例如

Option(p(0).trim.toInt) getOrElse 0

或者

Try(p(0).trim.toInt) getOrElse 0

这应该足以避免这样的投射错误

您在代码中提到case class仅支持22个元素。在较新版本的spark中已删除此限制。我希望答案有所帮助

编辑

：
注意： Option仅捕获NullPointerException，因此无法解决转换异常。 Try处理几乎所有类型的Exception，因此Try应该是投放例外的选项。

scala中的java.lang.NumberFormatException for mllib

1 个答案: