Question

我需要创建一个包含11个参数的Spark UDF。有没有办法实现呢？
我知道我们可以创建一个最多有10个参数的UDF

以下是10个参数的代码。它的工作原理

val testFunc1 = (one: String, two: String, three: String, four: String,
                 five: String, six: String, seven: String, eight: String, nine: String, ten: String) => {
    if (isEmpty(four)) false
    else four match {
        case "RDIS" => three == "ST"
        case "TTSC" => nine == "UT" && eight == "RR"
        case _ => false
    }
}
import org.apache.spark.sql.functions.udf    
udf(testFunc1)

以下是11个参数的代码。面对“未指定的值参数：dataType”问题

val testFunc2 = (one: String, two: String, three: String, four: String,
                 five: String, six: String, seven: String, eight: String, nine: String, ten: String, ELEVEN: String) => {
  if (isEmpty(four)) false
  else four match {
    case "RDIS" => three == "ST"
    case "TTSC" => nine == "UT" && eight == "RR" && ELEVEN == "OR"
    case _ => false
  }
}
import org.apache.spark.sql.functions.udf    
udf(testFunc2) // compilation error

Answer 1

我建议将参数打包到Map：

val df = sc.parallelize(Seq(("a","b"),("c","d"),("e","f"))).toDF("one","two")


val myUDF = udf((input:Map[String,String]) => {
  // do something with the input
  input("one")=="a"
})

df
  .withColumn("udf_args",map(
    lit("one"),$"one",
    lit("two"),$"one"
  )
 )
 .withColumn("udf_result", myUDF($"udf_args"))
 .show()

+---+---+--------------------+----------+
|one|two|            udf_args|udf_result|
+---+---+--------------------+----------+
|  a|  b|Map(one -> a, two...|      true|
|  c|  d|Map(one -> c, two...|     false|
|  e|  f|Map(one -> e, two...|     false|
+---+---+--------------------+----------+

Answer 2

您可以创建一个新列，该列是列数组：

df.withColumns("arrCol", array("col1", "col2", "col3", ...)

现在你可以做一个数组的UDF

val testFunc(vals: Seq[String]): String = ...

Spark Scala UDF参数限制为10

2 个答案: