Question

我已将url中的json数据保存到spark文件夹中名为urljson.json的json文件中。并已执行波纹管代码以在其上创建数据框通过这个

path="urljson.json/"
testdf1=spark.read.json(path)
testdf1.show()

我有这个

执行后

tesdf1.printSchema()

波纹管格式正在显示根 | - _corrupt_record：string（nullable = true）

我怎么能解决这个问题，我们将非常感谢任何指导我正在使用spark 2.0

我的json数据看起来很像我发布了它的一部分

result:[{"BldgID":"1006AVE ","BldgName":"100-6th Avenue SW (Oddfellows)          ","BldgCity":"Calgary             ","BldgState":"AB ","BldgZip":"T2G 2C4  ","BldgAddress1":"100-6th Avenue Southwest                ","BldgAddress2":"ZZZ None","BldgPhone":"4035439600     ","BldgLandlord":"1006AV","BldgLandlordName":"100-6 TH Avenue SW Inc.                                     ","BldgManager":"AVANDE","BldgManagerName":"Alyssa Van de Vorst           ","BldgManagerType":"Internal","BldgGLA":"34242","BldgEntityID":"1006AVE ","BldgInactive":"N","BldgPropType":"ZZZ None","BldgPropTypeDesc":"ZZZ None","BldgPropSubType":"ZZZ None","BldgPropSubTypeDesc":"ZZZ None","BldgRetailFlag":"N","BldgEntityType":"REIT                     ","BldgCityName":"Calgary             ","BldgDistrictName":"Downtown            ","BldgRegionName":"Western Canada                                    ","BldgAccountantID":"KKAUN     ","BldgAccountantName":"Kendra Kaun                   ","BldgAccountantMgrID":"LVALIANT  ","BldgAccountantMgrName":"Lorretta Valiant                        ","BldgFASBStartDate":"2012-10-24","BldgFASBStartDateStr":"2012-10-24"},{"BldgID":"1007AVE ","BldgName":"100-7th Avenue Southwest-Art Central    ","BldgCity":"Calgary             ","BldgState":"AB ","BldgZip":"T2P 0W4  ","BldgAddress1":"100-7th Avenue Southwest                ","BldgAddress2":"ZZZ None","BldgPhone":"4035439600     ","BldgLandlord":"1007AV","BldgLandlordName":"100-7th Avenue SW (Art Central) Inc.                        ","BldgManager":"LPATER","BldgManagerName":"Lyndsey Paterson              ","BldgManagerType":"Internal","BldgGLA":"27127","BldgEntityID":"1007AVE ","BldgInactive":"N","BldgPropType":"ZZZ None","BldgPropTypeDesc":"ZZZ None","BldgPropSubType":"ZZZ None","BldgPropSubTypeDesc":"ZZZ None","BldgRetailFlag":"N","BldgEntityType":"Property Under Dev't     ","BldgCityName":"Calgary             ","BldgDistrictName":"Downtown            ","BldgRegionName":"Western Canada                                    ","BldgAccountantID":"ABRITTON  ","BldgAccountantName":"Angie Britton                 ","BldgAccountantMgrID":"ZZZ None","BldgAccountantMgrName":"ZZZ None","BldgFASBStartDate":"2011-09-01","BldgFASBStartDateStr":"2011-09-01"},{"BldgID":"100LOMB ","BldgName":"100 Lombard Street                      ","BldgCity":"Toronto             ","BldgState":"ON ","BldgZip":"M5C 1M3  ","BldgAddress1":"100 Lombard Street                      ","BldgAddress2":"ZZZ None","BldgPhone":"4169779002     ","BldgLandlord":"100LOM","BldgLandlordName":"100 Lombard Street Inc.                                     ","BldgManager":"TCHALM","BldgManagerName":"Tiffany Chalmers              ","BldgManagerType":"Internal","BldgGLA":"43697.64","BldgEntityID":"100LOMB ","BldgInactive":"N","BldgPropType":"ZZZ None","BldgPropTypeDesc":"ZZZ None","BldgPropSubType":"ZZZ None","BldgPropSubTypeDesc":"ZZZ None","BldgRetailFlag":"N","BldgEntityType":"REIT                     ","BldgCityName":"Toronto             ","BldgDistrictName":"Queen - Richmond    ","BldgRegionName":"Central Canada                                    ","BldgAccountantID":"MALLORDE  ","BldgAccountantName":"May Ann Allorde               ","BldgAccountantMgrID":"TTSANG    ","BldgAccountantMgrName":"Tony Tsang                              ","BldgFASBStartDate":"2005-11-01","BldgFASBStartDateStr":"2005-11-01"},{"BldgID":"10190104","BldgName":"10190-104th Street NW-The Metals Buildi ","BldgCity":"Edmonton            ","BldgState":"AB ","BldgZip":"T5J 1A7  ","BldgAddress1":"10190-104st Street SW                   ","BldgAddress2":"ZZZ None","BldgPhone":"7804234400     ","BldgLandlord":"10190 ","BldgLandlordName":"10190-104 Street Inc.                                       ","BldgManager":"NEWWES","BldgManagerName":"New West Enterprise Property  ","BldgManagerType":"Third   ","BldgGLA":"20447.75","BldgEntityID":"10190104","BldgInactive":"N","BldgPropType":"ZZZ None","BldgPropTypeDesc":"ZZZ None","BldgPropSubType":"ZZZ None","BldgPropSubTypeDesc":"ZZZ None","BldgRetailFlag":"N","BldgEntityType":"REIT                     ","BldgCityName":"Edmonton            ","BldgDistrictName":"Edmonton            ","BldgRegionName":"Western Canada                                    ","BldgAccountantID":"RYANG     ","BldgAccountantName":"Raymond Yang                  ","BldgAccountantMgrID":"LVALIANT  ","BldgAccountantMgrName":"Lorretta Valiant                        ","BldgFASBStartDate":"2011-08-08","BldgFASBStartDateStr":"2011-08-08"}]

Answer 1

检查您在http://jsonlint.com/中提供的部分JSON导致错误：not a valid JSON。

从部分JSON中删除result:并检入http://jsonlint.com/，结果为valid JSON

请注意，在您的情况下，即使从JSON输入中删除“result：”，也可能不会产生值spark spark JSON输入，因为spark仅支持有限类型的JSON：

http://spark.apache.org/docs/latest/sql-programming-guide.html#json-datasets

JSON数据集

Spark SQL可以自动推断JSON数据集的架构并将其加载为数据集[Row]。可以在String的RDD或JSON文件上使用SparkSession.read.json（）完成此转换。

请注意，作为json文件提供的文件不是典型的JSON文件。每行必须包含一个单独的，自包含的有效JSON对象。有关更多信息，请参阅JSON Lines文本格式，也称为换行符分隔的JSON。因此，常规的多行JSON文件通常会失败。

在保存的json文件中创建数据框后，在pyspark中显示损坏的记录

1 个答案: