Question

我具有以下格式的数据，并且我想使用带有两列（“标签”和“数据”）的pyspark更改其格式。 “标签”列的值是唯一的，“数据”列的值是从原始列“日期，库存，价格”获得的json字符串。其中将“库存”和“价格”组合为“ A”列值，将“日期”和“数字”组合为“ B”列值。

我没有找到或写出好函数来实现这种效果。

我的Spark版本是2.1.0

原始DataFrame

date, stock, price， tag, num
1388534400, GOOG, 50, a, 1
1388534400, FB, 60, b, 2
1388534400, MSFT, 55, c, 3
1388620800, GOOG, 52, d, 4

我希望输出：

新的DataFrame

tag|                               data

'a'| "{'A':{'stock':'GOOD', 'price': 50}, B:{'date':1388534400, 'num':1}"

'b'| "{'A':{'stock':'FB', 'price': 60}, B:{'date':1388534400, 'num':2}"

'c'| "{'A':{'stock':'MSFT', 'price': 55}, B:{'date':1388534400, 'num':3}"

'd'| "{'A':{'stock':'GOOG', 'price': 52}, B:{'date':1388620800, 'num':4}"

+-+ ------------------------------------------- ------------------- +

from pyspark.sql import SparkSession
from pyspark.sql.functions import create_map
spark = SparkSession.builder.appName("example").getOrCreate()
df = spark.createDataFrame([
(1388534400, "GOOG", 50, 'a', 1),
(1388534400, "FB", 60, 'b', 2),
(1388534400, "MSFT", 55, 'c', 3),
(1388620800, "GOOG", 52, 'd', 4)]
).toDF("date", "stock", "price", 'tag', 'num')
df.show()
tag_cols = {'A':['stock', 'price'], 'B':['date', 'num']}

# todo, change the Dataframe columns format

Answer 1

IIUC，只需使用pyspark.sql.functions.struct和pyspark.sql.functions.to_json（两者都应在spark 2.1中使用）

from pyspark.sql import functions as F

# skip df initialization[enter link description here][1]

df_new = df.withColumn('A', F.struct('stock', 'price')) \
           .withColumn('B', F.struct('date', 'num')) \
           .select('tag', F.to_json(F.struct('A', 'B')).alias('data'))

>>> df_new.show(5,0)
+---+-----------------------------------------------------------------+
|tag|data                                                             |
+---+-----------------------------------------------------------------+
|a  |{"A":{"stock":"GOOG","price":50},"B":{"date":1388534400,"num":1}}|
|b  |{"A":{"stock":"FB","price":60},"B":{"date":1388534400,"num":2}}  |
|c  |{"A":{"stock":"MSFT","price":55},"B":{"date":1388534400,"num":3}}|
|d  |{"A":{"stock":"GOOG","price":52},"B":{"date":1388620800,"num":4}}|
+---+-----------------------------------------------------------------+

如何将一些pyspark数据帧的列转换为具有其列名的字典，并将其组合为json列？

1 个答案: