将字符串转换为Pyspark数据帧

Question

我在列表中有一个字符串，例如

ListofString = ['Column1,Column2,Column3,\nCol1Value1,Col2Value1,Col3Value1,\nCol1Value2,Col2Value2,Col3Value2']

如何将该字符串转换为如下所示的pyspark Dataframe

'\ n'为新行

Column1         Column2         Column3
-----------------------------------------
Col1Value1      Col2Value1      Col3Value1
Col1Value2      Col2Value2      Col3Value2

Answer 1

您只需要按照以下正确格式转换字符串列表：

# convert the list of string into proper format
>>> l = ' '.join(ListofString)
>>> l = l.replace(',',' ')
>>> l = [x.strip().split(' ') for x in l.split('\n')]

>>> print(l)

>>> [['Column1', 'Column2', 'Column3'], ['Col1Value1', 'Col2Value1', 'Col3Value1'], ['Col1Value2', 'Col2Value2', 'Col3Value2']]

>>> df = spark.createDataFrame(l[1:],l[0])

>>> df.show()

+----------+----------+----------+
|   Column1|   Column2|   Column3|
+----------+----------+----------+
|Col1Value1|Col2Value1|Col3Value1|
|Col1Value2|Col2Value2|Col3Value2|
+----------+----------+----------+