Question

有人可以告诉我如何将包含字符串的列表转换为pyspark中的Dataframe。我使用python 3.6与spark 2.2.1。我刚刚开始学习spark环境，我的数据如下所示

my_data =[['apple','ball','ballon'],['cat','camel','james'],['none','focus','cake']]

现在，我想创建一个Dataframe，如下所示

---------------------------------
|ID | words                     |
---------------------------------
 1  | ['apple','ball','ballon'] |
 2  | ['cat','camel','james']   |

我甚至想添加数据

中没有关联的ID列

Answer 1

您可以将列表转换为Row对象列表，然后使用spark.createDataFrame来推断数据中的架构：

from pyspark.sql import Row
R = Row('ID', 'words')

# use enumerate to add the ID column
spark.createDataFrame([R(i, x) for i, x in enumerate(my_data)]).show() 
+---+--------------------+
| ID|               words|
+---+--------------------+
|  0|[apple, ball, bal...|
|  1| [cat, camel, james]|
|  2| [none, focus, cake]|
+---+--------------------+

Answer 2

试试这个 -

data_array = []
for i in range (0,len(my_data)) :
    data_array.extend([(i, my_data[i])])

df = spark.createDataframe(data = data_array, schema = ["ID", "words"])

df.show()

Answer 3

Try this -- the simplest approach

  from pyspark.sql import *
  x = Row(utc_timestamp=utc, routine='routine name', message='your message')
  data = [x]
  df = sqlContext.createDataFrame(data)

Answer 4

简单方法：

npx --ignore-existing react-native init MyApp --template react-native-template-typescript

在pyspark中列出到DataFrame

4 个答案: