如何使用天蓝色的pyspark数据框将具有空值的数据写入json文件?

时间:2020-01-24 23:23:04

标签: azure dataframe apache-spark pyspark azure-databricks

我正在使用以下方式将数据写入json文件,但是具有Null值的列将被忽略:

dfAllData.write.format("json").save("/mnt/coi/DataModel")

#Merge Files
outfilename = "/dbfs/mnt/coi/FinalData/FinalData.json"
with open(outfilename, 'wb') as outfile:
    for filename in glob.glob("/dbfs/mnt/coi/DataModel/part-0*.json"):
        if filename == outfilename:
            # don't want to copy the output into the output
            continue
        with open(filename, 'rb') as readfile:
            shutil.copyfileobj(readfile, outfile)

是否有设置写入空值的任何选项。 ?

0 个答案:

没有答案