我有一个如下所示的数据框
| id| age| rbc| bgr| dm|cad|appet| pe|ane|classification|
+---+----+------+-----+---+---+-----+---+---+--------------+
| 3|48.0|normal|117.0| no| no| poor|yes|yes| ckd|
....
....
....
我编写了一个UDF,将分类yes, no, poor, normal
转换为二进制0s
和1s
def stringToBinary(stringValue: String): Int = {
stringValue match {
case "yes" => return 1
case "no" => return 0
case "present" => return 1
case "notpresent" => return 0
case "normal" => return 1
case "abnormal" => return 0
}
}
val stringToBinaryUDF = udf(stringToBinary _)
我将此应用于数据框,如下所示
val newCol = stringToBinaryUDF.apply(col("pc")) //creates the new column with formatted value
val refined1 = noZeroDF.withColumn("dm", newCol) //adds the new column to original
如何将多个列传递到UDF中,以便我不必为其他分类列重复自己?
答案 0 :(得分:7)
udf
函数执行与spark
函数相同的工作,那么 udf
函数不应该是选择,函数会对列数据进行序列化和反序列化。
将dataframe
视为
+---+----+------+-----+---+---+-----+---+---+--------------+
|id |age |rbc |bgr |dm |cad|appet|pe |ane|classification|
+---+----+------+-----+---+---+-----+---+---+--------------+
|3 |48.0|normal|117.0|no |no |poor |yes|yes|ckd |
+---+----+------+-----+---+---+-----+---+---+--------------+
您可以使用when
函数
import org.apache.spark.sql.functions._
def applyFunction(column : Column) = when(column === "yes" || column === "present" || column === "normal", lit(1))
.otherwise(when(column === "no" || column === "notpresent" || column === "abnormal", lit(0)).otherwise(column))
df.withColumn("dm", applyFunction(col("dm")))
.withColumn("cad", applyFunction(col("cad")))
.withColumn("rbc", applyFunction(col("rbc")))
.withColumn("pe", applyFunction(col("pe")))
.withColumn("ane", applyFunction(col("ane")))
.show(false)
结果是
+---+----+---+-----+---+---+-----+---+---+--------------+
|id |age |rbc|bgr |dm |cad|appet|pe |ane|classification|
+---+----+---+-----+---+---+-----+---+---+--------------+
|3 |48.0|1 |117.0|0 |0 |poor |1 |1 |ckd |
+---+----+---+-----+---+---+-----+---+---+--------------+
现在问题清楚地表明,您不想重复所有列的过程,您可以执行以下操作
val columnsTomap = df.select("rbc", "cad", "rbc", "pe", "ane").columns
var tempdf = df
columnsTomap.map(column => {
tempdf = tempdf.withColumn(column, applyFunction(col(column)))
})
tempdf.show(false)
答案 1 :(得分:1)
UDF 可以使用许多参数,即许多列,但它应返回一个结果,即一列。
为了做到这一点,只需在stringToBinary
功能中添加参数即可。
您希望它采用两列,如下所示:
def stringToBinary(stringValue: String, secondValue: String): Int = {
stringValue match {
case "yes" => return 1
case "no" => return 0
case "present" => return 1
case "notpresent" => return 0
case "normal" => return 1
case "abnormal" => return 0
}
}
val stringToBinaryUDF = udf(stringToBinary _)
希望这有帮助
答案 2 :(得分:0)
您还可以使用foldLeft
函数。将您的 UDF 称为stringToBinaryUDF
:
import org.apache.spark.sql.functions._
val categoricalColumns = Seq("rbc", "cad", "rbc", "pe", "ane")
val refinedDF = categoricalColumns
.foldLeft(noZeroDF) { (accumulatorDF: DataFrame, columnName: String) =>
accumulatorDF
.withColumn(columnName, stringToBinaryUDF(col(columnName)))
}
这将尊重不变性和功能性编程。