假设我有一个数据框:
df = sqlContext.createDataFrame(
[(1, 10, 21.0, 0), (3, 14, -23.0, 1)], ("x1", "x2", "x3", "x4"))
df.show()
## +---+---+-----+---+
## | x1| x2| x3| x4|
## +---+---+-----+---+
## | 1| 10| 23.0| 5|
## | 3| 14|-23.0| 0|
## +---+---+-----+---+
在这些重复项中“复制”行并设置x4=1
的有效方法是什么?
## +---+---+-----+---+
## | x1| x2| x3| x4|
## +---+---+-----+---+
## | 1| 10| 23.0| 5|
## | 1| 10| 23.0| 1|
## | 3| 14|-23.0| 0|
## | 3| 14|-23.0| 1|
## +---+---+-----+---+
在Apache PIG中,模拟很简单:做一个foreach并生成:
FLATTEN(TOBAG(1, x4)) AS x4
谢谢大家
答案 0 :(得分:4)
从pyspark.sql.functions
导入所需的功能:
from pyspark.sql.functions import array, explode, lit
并替换现有列:
df.withColumn("x4", explode(array(lit(1), df["x4"])))