Question

考虑以下数据框，

cols = ['Id', 'col1', 'col2', 'col3']
vals = [('1A','Not his side|:|This side', 'This side', 'Not this either|:|but this'),  
        ('1B','Keep this', 'This one|:|keep this', 'remove|:|keep that')]

dd1 = sqlContext.createDataFrame(vals, cols)

#+---+------------------------+--------------------+--------------------------+
#|Id |col1                    |col2                |col3                      |
#+---+------------------------+--------------------+--------------------------+
#|1A |Not his side|:|This side|This side           |Not this either|:|but this|
#|1B |Keep this               |This one|:|keep this|remove|:|keep that        |
#+---+------------------------+--------------------+--------------------------+

我需要做的是在|:|处分割字符串并保留第二个单词。但是，如果字符串不包含定界符（|:|），那么我得到null，即：

users1 = [F.split(F.col(x), "\\|:\\|").alias(x) for x in cols]
dd1.select(*users1).show()

#+---------+---------+---------+
#|     col1|     col2|     col3|
#+---------+---------+---------+
#|This side|     null| but this|
#|     null|keep this|keep that|
#+---------+---------+---------+

我正在寻找的结果是：

+---+---------+---------+---------+
|Id |col1     |col2     |col3     |
+---+---------+---------+---------+
|1A |This side|This side|but this |
|1B |Keep this|keep this|keep that|
+---+---------+---------+---------+

Answer 1

使用when和otherwise并检查字符串是否包含"|:|"。可以完成以下操作：

cols = ['col1', 'col2', 'col3']

users1 = [F.when(F.col(x).contains("|:|"), F.split(F.col(x), "\\|:\\|")[1]).otherwise(F.col(x)).alias(x) for x in cols]
dd1.select(F.col('Id'), *users1)

此处cols仅包含要拆分的列。最后的select也将包含Id列。

Answer 2

您可以将when和size内置函数用作

users1 = [F.when(F.size(F.split(F.col(x), "\\|:\\|")) > 1, F.split(F.col(x), "\\|:\\|")[1]).otherwise(F.col(x)).alias(x) for x in cols]
dd1.select(*users1).show()

应该给您

+---+---------+---------+---------+
| Id|     col1|     col2|     col3|
+---+---------+---------+---------+
| 1A|This side|This side| but this|
| 1B|Keep this|keep this|keep that|
+---+---------+---------+---------+

您可以修改答案，以便只能使用split函数一次。

我希望答案是有帮助的，应该是如何进行的很好的提示。

Answer 3

您在问题中定义的选择不会产生您提到的结果。我想您忘记了从数组中选择元素；）

users1 = [F.split(F.col(x), "\\|:\\|").alias(x) for x in cols]
dd1.select(*users1).show()
+----+--------------------+--------------------+--------------------+
|  Id|                col1|                col2|                col3|
+----+--------------------+--------------------+--------------------+
|[1A]|[Not his side, Th...|         [This side]|[Not this either,...|
|[1B]|         [Keep this]|[This one, keep t...| [remove, keep that]|
+----+--------------------+--------------------+--------------------+

要实现所需的功能，只需选择数组的最后一个元素。您可以为此使用size函数：

users2 = [F.col(x).getItem(F.size(F.col(x))-1).alias(x) for x in cols]
dd1.select(*users1).select(*users2).show()
+---+---------+---------+---------+
| Id|     col1|     col2|     col3|
+---+---------+---------+---------+
| 1A|This side|This side| but this|
| 1B|Keep this|keep this|keep that|
+---+---------+---------+---------+

找到分割字符串IF分隔符

3 个答案: