Question

我目前正在尝试将json作为熊猫数据帧处理。这里发生的是，我得到了连续的json结构流。它们只是简单地附加。这是一整行。我从中提取了一个.txt，现在想通过熊猫对其进行分析。

示例片段：

{“ positionFlightMessage”：{“ messageUuid”：“ 95b3b6ca-5dd2-44b4-918a-baa51022d143”，“ schemaVersion”：“ 1.0-RC1”，“时间戳”：1533134514，“ flightNumber”：“ DLH1601”，“位置”：{“航点”：{“纬度”：44.14525，“经度”：-1.31849}，“ flightLevel”：340，“标题”：24.0}，“ messageSource”：“ ADSB”，“ flightUniqueId”：“ AFR1601 -1532928365-airline-0002“，” airlineIcaoCode“：” AFR“，” atcCallsign“：” AFR89GA“，”燃料“：{}，”速度“：{” groundSpeed“：442.0}，” altitude“：{” altitude “：34000.0}，” nextPosition“：{” waypoint“：{}}，” messageSubtype“：” ADSB“}} {” positionFlightMessage“：{” messageUuid“：” 884708c1-2fff-4ebf-b72c-bbc6ed2c3623“，” schemaVersion“：” 1.0-RC1“，”时间戳“：1533134515，” flightNumber“：” DLH012“，”位置“：{”航点“：{”纬度“：37.34542，”经度“：143.79951}，” flightLevel“： 320，“标题”：54.0}，“ messageSource”：“ ADSB”，“ flightUniqueId”：“ EVA12-1532928367-airline-0096”，“ airlineIcaoCode”：“ DLH”，“ atcCallsign”：“ EVA012”，“ fuel” ：{}，“ speed”：{“ groundSpeed”：462.0}，“ altitude”：{“ altitude”：32000.0}，“ nextPosition”：{“ waypoint”：{}}，“ messageSubtype”：“ A DSB“}} ...

从这个角度您可以看到，每个json以{"positionFlightMessage":开头并以messageSubtype":"ADSB"结尾

一个json结束后，下一个json会追加到它后面。

我需要的是一张桌子，像这样：

95b3b6ca-5dd2-44b4-918a-baa51022d143    1.0-RC1 1533134514  DLH1601 4.414.525   -131.849    340 24.0    ADSB    AFR1601-1532928365-airline-0002 AFR AFR89GA 442.0   34000.0 ADSB
884708c1-2fff-4ebf-b72c-bbc6ed2c3623    1.0-RC1 1533134515  DLH012  3.734.542   14.379.951  320 54.0    ADSB    EVA12-1532928367-airline-0096   DLH EVA012  462.0   32000.0 ADSB

我尝试使用熊猫读取json，但出现错误。

import pandas as pd

df = pd.read_json("tD.txt",orient='columns')

df.head()

ValueError：尾随数据

tD.txt具有上面给出的代码段，没有最后一个（...）点

我认为问题是，每个json都只是附加了。我可以在每行之后添加新行

messageSubtype":"ADSB"}}

然后阅读它，但是也许您有一种解决方案，我可以直接将大txt文件转换为df并轻松将其转换为dp

Answer 1

尝试获取要输出的json流，如下所示：

注意开始的'['和结束的']'。还要注意每个json输入之间的'，'。

data = [{
    "positionFlightMessage": {
        "messageUuid": "95b3b6ca-5dd2-44b4-918a-baa51022d143",
        "schemaVersion": "1.0-RC1",
        "timestamp": 1533134514,
        "flightNumber": "DLH1601",
        "position": {
            "waypoint": {
                "latitude": 44.14525,
                "longitude": -1.31849
            },
            "flightLevel": 340,
            "heading": 24.0
        },
        "messageSource": "ADSB",
        "flightUniqueId": "AFR1601-1532928365-airline-0002",
        "airlineIcaoCode": "AFR",
        "atcCallsign": "AFR89GA",
        "fuel": {},
        "speed": {
            "groundSpeed": 442.0
        },
        "altitude": {
            "altitude": 34000.0
        },
        "nextPosition": {
            "waypoint": {}
        },
        "messageSubtype": "ADSB"
    }
}, {
    "positionFlightMessage": {
        "messageUuid": "884708c1-2fff-4ebf-b72c-bbc6ed2c3623",
        "schemaVersion": "1.0-RC1",
        "timestamp": 1533134515,
        "flightNumber": "DLH012",
        "position": {
            "waypoint": {
                "latitude": 37.34542,
                "longitude": 143.79951
            },
            "flightLevel": 320,
            "heading": 54.0
        },
        "messageSource": "ADSB",
        "flightUniqueId": "EVA12-1532928367-airline-0096",
        "airlineIcaoCode": "DLH",
        "atcCallsign": "EVA012",
        "fuel": {},
        "speed": {
            "groundSpeed": 462.0
        },
        "altitude": {
            "altitude": 32000.0
        },
        "nextPosition": {
            "waypoint": {}
        },
        "messageSubtype": "ADSB"
    }
}]

现在，您应该可以遍历json中的每个“列表”元素，并将其附加到pandas df。

print(len(data))
for i in range(0,len(data)):
    #here is just show messageSource only. Up to you to find out the rest..
    print(data[i]['positionFlightMessage']['messageSource'])
    #instead of printing here you should append it to pandas df.

希望这对您有所帮助。

Answer 2

现在这是使用正则表达式的JSON解决方案。

s = '{"positionFlightMessage":{"messageUuid":"95b3b6ca-5dd2-44b4-918a-baa51022d143","schemaVersion":"1.0-RC1","timestamp":1533134514,"flightNumber":"DLH1601","position":{"waypoint":{"latitude":44.14525,"longitude":-1.31849},"flightLevel":340,"heading":24.0},"messageSource":"ADSB","flightUniqueId":"AFR1601-1532928365-airline-0002","airlineIcaoCode":"AFR","atcCallsign":"AFR89GA","fuel":{},"speed":{"groundSpeed":442.0},"altitude":{"altitude":34000.0},"nextPosition":{"waypoint":{}},"messageSubtype":"ADSB"}}{"positionFlightMessage":{"messageUuid":"884708c1-2fff-4ebf-b72c-bbc6ed2c3623","schemaVersion":"1.0-RC1","timestamp":1533134515,"flightNumber":"DLH012","position":{"waypoint":{"latitude":37.34542,"longitude":143.79951},"flightLevel":320,"heading":54.0},"messageSource":"ADSB","flightUniqueId":"EVA12-1532928367-airline-0096","airlineIcaoCode":"DLH","atcCallsign":"EVA012","fuel":{},"speed":{"groundSpeed":462.0},"altitude":{"altitude":32000.0},"nextPosition":{"waypoint":{}},"messageSubtype":"ADSB"}}'

import re
import json
replaced = json.loads('['+re.sub(r'{\"positionFlightMessage*', ',{\"positionFlightMessage', s)[1:] + ']')

dfTemp = pd.DataFrame(data=replaced)
df = pd.DataFrame()
counter = 0
def newDf(row):
  global df,counter
  counter += 1
  temp = pd.DataFrame([row])
  df = df.append(temp)
dfTemp['positionFlightMessage'] = dfTemp['positionFlightMessage'].apply(newDf)
print(df)

首先，我们将所有出现的{"positionFlightMessage替换为,{"positionFlightMessage，并丢弃第一个分隔符。
我们以此创建一个数据框，但是这里只有一列。使用列上的apply函数，并从中创建一个新的数据框。
从此数据框中，您可以执行更多清理工作。

JSON提取到熊猫数据框

2 个答案: