Question

我尝试生成一个JSON字符串，以便在一个STRING列中存储可变数量的历史记录。代码适用于我的所有小测试，但在实际数据上运行时失败（没有错误，只有没有数据）。这就是我所拥有的：

class HistoryDetail (
  var date : String,
  var val1 : Int,
  var val2 : Int,
  var changeCode : String
)

class HistoryHeader(
  var numDetailRecords : Int,
  var calcDate : String,
  var historyRecords : List[HistoryDetail]
)

def getJSON = (val1:Int, val2:Int) => {
  implicit val formats = org.json4s.DefaultFormats;
  val today = LocalDate.now
  val hdl = List(new HistoryDetail(today.toString, val1, val2, "D"))
  val hh:HistoryHeader = new HistoryHeader(1,today.toString,hdl)
  Serialization.write(hh);
}

调用Scala函数的非常简单的测试工作正常：

val strJson = getJSON(1000,1000)
strJson: String = {"numDetailRecords":1,"calcDate":"2018-04-23","historyRecords":[{"date":"2018-04-23","val1":1000,"val2":1000,"changeCode":"D"}]}

创建UDF并应用于小型DataFrame可以正常工作：

spark.udf.register("getJSONUDF", getJSON)
val smallDF = Seq((100, 100), (101, 101), (102, 102)).toDF("int_col1", "int_col2").withColumn("json_col",callUDF("getJSONUDF", $"int_col1", $"int_col2"))
smallDF.show(false)

+--------+--------+------------------------------------------------------------------------------------------------------------------------------+
|int_col1|int_col2|json_col                                                                                                                      |
+--------+--------+------------------------------------------------------------------------------------------------------------------------------+
|100     |100     |{"numDetailRecords":1,"calcDate":"2018-04-23","historyRecords":[{"date":"2018-04-23","val1":100,"val2":100,"changeCode":"D"}]}|
|101     |101     |{"numDetailRecords":1,"calcDate":"2018-04-23","historyRecords":[{"date":"2018-04-23","val1":101,"val2":101,"changeCode":"D"}]}|
|102     |102     |{"numDetailRecords":1,"calcDate":"2018-04-23","historyRecords":[{"date":"2018-04-23","val1":102,"val2":102,"changeCode":"D"}]}|
+--------+--------+------------------------------------------------------------------------------------------------------------------------------+

针对实际数据运行它失败（再次没有错误，只是没有数据）：

val bigDF = spark.read.table("table_name")
                      .select($"int_col1",$"int_col2")
                      .withColumn("json_col",callUDF("getJSONUDF", $"int_col1", $"int_col2"))
bigDF.show(false)
+--------+--------+--------+
|int_col1|int_col2|json_col|
+--------+--------+--------+
|18995   |12702   |{}      |
|14989   |46998   |{}      |
|25588   |25051   |{}      |
|18750   |52282   |{}      |
|19963   |25745   |{}      |
|17500   |21587   |{}      |
|21999   |20379   |{}      |
|25975   |5988    |{}      |
|26382   |5988    |{}      |
|7049    |101907  |{}      |
|45997   |47472   |{}      |
|45997   |47472   |{}      |
|13950   |158957  |{}      |
|18999   |123689  |{}      |
|33842   |69623   |{}      |
|64000   |13362   |{}      |
|64000   |13362   |{}      |
|64000   |13362   |{}      |
|64000   |13362   |{}      |
|64000   |13362   |{}      |
+--------+--------+--------+
only showing top 20 rows

（版本：java 1.8.0_60，spark 2.2.0，scala 2.11.8）

有关为什么在使用较大的DataFrame时获取空JSON对象的任何想法？

Answer 1

TBH我不知道到底出了什么问题，因为我原本预计会出现Task not serializable错误。我最好的猜测是getJSON不是线程安全的，并且在要编码的对象的某个地方保持集中状态。再次，这可能是完全错误的

我认为通过使用Spark的to_json函数，您的方法可能会更好。

def getHistoryReader = udf((val1:Int, val2:Int) => {
  val today = LocalDate.now
  val hdl = List(new HistoryDetail(today.toString, val1, val2, "D"))
  new HistoryHeader(1,today.toString,hdl)
})

val bigDF = spark.read.table("table_name")
  .select($"int_col1",$"int_col2")
  .withColumn("json_col",to_json(getHistoryReader($"int_col1", $"int_col2")))

bigDF.show(false)

在我看来看起来更干净，并将json序列化留给了Spark。这应该工作

UDF生成JSON字符串行为不一致

1 个答案: