Pyspark 将 StrucType[ArrayType, StructureType] 转换为 ArrayType[ArrayType,ArrayType]

时间:2021-01-08 09:21:01

标签: dataframe pyspark user-defined-functions

我有一个 DF,列 'COL_NAME' 包含一个 StructType 格式:

StructType[StructType, StructType, ArrayType, ArrayType, StructType, ArrayType, ArrayType]

上面从 UDF 返回的 StructType。我需要将其强制转换为所有 ArrayType

ArrayType[ArrayType, ArrayType, ArrayType, ArrayType, ArrayType, ArrayType, ArrayType]

怎么做。非常感谢

更新

示例:

array need to return :
[[before], [after], [from_tbl], [where_tbl], [to_tbl], [lst_tbl], [dbinfo]]

where before/after is a list
before, after = [[query_type,out,[from],[where]],[query_type,out,[from],[where]]]

where querytype is string, out is string, but from, where is list, not same level

to_tbl = [write_mode, [table_name], table_action]
from_tbl, where_tbl, from, where, table_name, lst_tbl, db_info = [a,b,c]

当我定义 UDF 时,我必须定义返回模式,嵌套数组不是同一级别,因此我无法为所有人定义有效模式

0 个答案:

没有答案