Question

我正在尝试使用Scala Luna Eclipse IDE注册一个简单的UDF来提取spark中的日期功能。这是我的代码：
sqlContext.udf.register("extract", (dateUnit: String, date : String) => udf.extract(dateUnit,date ) )

 def extract(dateUnit : String, date: String) :  String = {
    val splitArray : Array[String] = date.split("-") 
        val result  = dateUnit.toUpperCase() match {
      case "YEAR" => splitArray(0)
      case "MONTH" => splitArray(1)
      case "DAY" => splitArray(2)
      case whoa => "Unexpected case :" + whoa.toString()
    }
    return result ;
  }

当我通过Eclipse控制台执行此功能时作为Select * from date_dim WHERE d_dom < extract('YEAR', '2015-05-01') limit 10"

它会抛出ans错误

org.apache.spark.sql.catalyst.errors.package$TreeNodeException: execute, tree:
Aggregate false, [], [Coalesce(SUM(PartialCount#30L),0) AS count#28L]
 Aggregate true, [], [COUNT(1) AS PartialCount#30L]
  Project []

    Caused by: org.apache.spark.SparkException: Task not serializable

    Caused by: java.lang.reflect.InvocationTargetException

    Caused by: java.lang.ArrayIndexOutOfBoundsException: 0 
    at java.io.ObjectStreamClass$FieldReflector.getObjFieldValues(ObjectStreamClass.java:2030)
        at java.io.ObjectStreamClass.getObjFieldValues(ObjectStreamClass.java:1232)

我无法找出究竟是什么问题，直接定义的简单udf如sqlContext.udf.register("strLength", (str: String) => str.length() ) 运行成功。上面的相同功能通过spark中的Scala shell成功运行。这里有什么问题。我做错了吗？

org.apache.spark.sql.catalyst.errors.package $ TreeNodeException：execute，tree：

0 个答案: