Question

我正在构建一个Python脚本，我需要从json RDD生成一个json文件。以下是用于保存json文件的代码片段。

jsonRDD.map(lambda x :json.loads(x))
.coalesce(1, shuffle=True).saveAsTextFile('examples/src/main/resources/demo.json')

但是我需要将json数据写入单个文件而不是分布在多个分区上的数据。

所以请为我建议适当的解决方案

Answer 1

Without the use of additional libraries like pandas, you could save your RDD of several jsons by reducing them to one big string of jsons, each separated by a new line:

# perform your operation
# note that you do not need a lambda expression for json.loads
jsonRDD = jsonRDD.map(json.loads).coalesce(1, shuffle=True)

# map jsons back to string
jsonRDD = jsonRDD.map(json.dumps)

# reduce to one big string with one json on each line
json_string = jsonRDD.reduce(lambda x, y: x + "\n" + y)

# write your string to a file
with open("path/to/your.json", "w") as f:
    f.write(json_string.encode("utf-8"))

Answer 2

一旦我将它们放在RDD或数据帧中，我就遇到了pyspark保存JSON文件的问题，所以我要做的就是将它们转换为pandas数据帧并将它们保存到非分布式目录。

import pandas

df1 = sqlContext.createDataFrame(yourRDD)
df2 = df1.toPandas()
df2.to_json(yourpath)

为pyspark RDD生成单个json文件

2 个答案: