我想验证数组在Pyspark中是否包含字符串(Spark <2.4)。
示例数据框:
column_1 <Array> | column_2 <String>
--------------------------------------------
["2345","98756","8794"] | 8794
--------------------------------------------
["8756","45678","987563"] | 1234
--------------------------------------------
["3475","8956","45678"] | 3475
--------------------------------------------
我想比较column_1和column_2两列。如果column_1包含column_2,则应从column_1跳过它的值。 我做了udf来从column_1插入column_2,但是没有用:
def contains(x, y):
try:
sx, sy = set(x), set(y)
if len(sx) == 0:
return sx
elif len(sy) == 0:
return sx
else:
return sx - sy
# in exception, for example `x` or `y` is None (not a list)
except:
return sx
udf_contains = udf(contains, 'string')
new_df = my_df.withColumn('column_1', udf_contains(my_df.column_1, my_df.column_2))
预期结果:
column_1 <Array> | column_2 <String>
--------------------------------------------------
["2345","98756"] | 8794
--------------------------------------------------
["8756","45678","987563"] | 1234
--------------------------------------------------
["8956","45678"] | 3475
--------------------------------------------------
我该怎么办?知道有时候/我拥有column_1的情况是[]而column_2为null?谢谢
答案 0 :(得分:2)
Spark 2.4.0 +
尝试array_remove
。自spark 2.4.0起可用:
val df = Seq(
(Seq("2345","98756","8794"), "8794"),
(Seq("8756","45678","987563"), "1234"),
(Seq("3475","8956","45678"), "3475"),
(Seq(), "empty"),
(null, "null")
).toDF("column_1", "column_2")
df.show(5, false)
df
.select(
$"column_1",
$"column_2",
array_remove($"column_1", $"column_2") as "diff"
).show(5, false)
它将返回:
+---------------------+--------+
|column_1 |column_2|
+---------------------+--------+
|[2345, 98756, 8794] |8794 |
|[8756, 45678, 987563]|1234 |
|[3475, 8956, 45678] |3475 |
|[] |empty |
|null |null |
+---------------------+--------+
+---------------------+--------+---------------------+
|column_1 |column_2|diff |
+---------------------+--------+---------------------+
|[2345, 98756, 8794] |8794 |[2345, 98756] |
|[8756, 45678, 987563]|1234 |[8756, 45678, 987563]|
|[3475, 8956, 45678] |3475 |[8956, 45678] |
|[] |empty |[] |
|null |null |null |
+---------------------+--------+---------------------+
对不起,我想用pyspark做到这一点很容易。
火花<2.4.0
%pyspark
from pyspark.sql.functions import udf
from pyspark.sql.types import ArrayType, StringType
data = [
(["2345","98756","8794"], "8794"),
(["8756","45678","987563"], "1234"),
(["3475","8956","45678"], "3475"),
([], "empty"),
(None,"null")
]
df = spark.createDataFrame(data, ['column_1', 'column_2'])
df.printSchema()
df.show(5, False)
def contains(x, y):
if x is None or y is None:
return x
else:
sx, sy = set(x), set([y])
return list(sx - sy)
udf_contains = udf(contains, ArrayType(StringType()))
df.select("column_1", "column_2", udf_contains("column_1", "column_2")).show(5, False)
结果:
root
|-- column_1: array (nullable = true)
| |-- element: string (containsNull = true)
|-- column_2: string (nullable = true)
+---------------------+--------+
|column_1 |column_2|
+---------------------+--------+
|[2345, 98756, 8794] |8794 |
|[8756, 45678, 987563]|1234 |
|[3475, 8956, 45678] |3475 |
|[] |empty |
|null |null |
+---------------------+--------+
+---------------------+--------+----------------------------+
|column_1 |column_2|contains(column_1, column_2)|
+---------------------+--------+----------------------------+
|[2345, 98756, 8794] |8794 |[2345, 98756] |
|[8756, 45678, 987563]|1234 |[8756, 987563, 45678] |
|[3475, 8956, 45678] |3475 |[8956, 45678] |
|[] |empty |[] |
|null |null |null |
+---------------------+--------+----------------------------+