我如何计算和跟踪巨大的json文件中的值

时间:2019-03-27 04:29:38

标签: python json data-structures

我有一个巨大的json文件,它的键调用类型(犯罪的类型),日期和时间(犯罪的日期)以及位置(地址或经纬度)以及其他带有值的键。我最感兴趣的是统计犯罪次数最多的日子,统计显示最多的呼叫类型以及显示最多的位置,该位置可以通过家庭住址或纬度和经度配对来衡量。 Python可能是最好的。 JSON上有350多种数据行的350多种呼叫类型。因此,每当您看到新的呼叫类型时,都应该为其创建一个新变量并对其进行跟踪

我尝试迭代将其扔进一个列表,但遇到了问题。当我的代码达到62 mb时,如何附加到我的代码上,我应该链接到文件吗?

这是数据示例

[{"A": "incident_num", "B": "date_time", "C": "day", "D": "stno", "E": "stdir1", "F": "StreetName", "G": "streettype", "H": "FullAddress", "I": "call_type", "J": "disposition", "K": "beat", "L": "priority", "M": "lat", "N": "long"},
{"A": "P17060024503", "B": "6/14/2017 21:54", "C": "4", "D": "10", "E": "", "F": "14TH", "G": "ST", "H": "10 14TH ST, San Diego, CA", "I": "1151", "J": "O", "K": "521", "L": "2", "M": "32.7054489", "N": "-117.1518696"},
{"A": "P17030051227", "B": "3/29/2017 22:24", "C": "4", "D": "10", "E": "", "F": "14TH", "G": "ST", "H": "10 14TH ST, San Diego, CA", "I": "1016", "J": "A", "K": "521", "L": "2", "M": "32.7054489", "N": "-117.1518696"},
{"A": "P17060004814", "B": "6/3/2017 18:04", "C": "7", "D": "10", "E": "", "F": "14TH", "G": "ST", "H": "10 14TH ST, San Diego, CA", "I": "1016", "J": "A", "K": "521", "L": "2", "M": "32.7054489", "N": "-117.1518696"},
{"A": "P17030029336", "B": "3/17/2017 10:57", "C": "6", "D": "10", "E": "", "F": "14TH", "G": "ST", "H": "10 14TH ST, San Diego, CA", "I": "1151", "J": "OT", "K": "521", "L": "2", "M": "32.7054489", "N": "-117.1518696"},
{"A": "P17030005412", "B": "3/3/2017 23:45", "C": "6", "D": "10", "E": "", "F": "15TH", "G": "ST", "H": "10 15TH ST, San Diego, CA", "I": "911P", "J": "CAN", "K": "521", "L": "2", "M": "32.7057215", "N": "-117.1503498"},
{"A": "P17020016091", "B": "2/10/2017 8:23", "C": "6", "D": "10", "E": "", "F": "15TH", "G": "ST", "H": "10 15TH ST, San Diego, CA", "I": "AU2", "J": "W", "K": "521", "L": "2", "M": "32.7057215", "N": "-117.1503498"},
{"A": "P17040017368", "B": "4/11/2017 4:57", "C": "3", "D": "10", "E": "", "F": "15TH", "G": "ST", "H": "10 15TH ST, San Diego, CA", "I": "5150", "J": "CAN", "K": "521", "L": "2", "M": "32.7057215", "N": "-117.1503498"},
{"A": "P17030048050", "B": "3/28/2017 6:30", "C": "3", "D": "10", "E": "", "F": "15TH", "G": "ST", "H": "10 15TH ST, San Diego, CA", "I": "1146", "J": "K", "K": "521", "L": "", "M": "32.7057215", "N": "-117.1503498"},
{"A": "P17060037341", "B": "6/22/2017 10:19", "C": "5", "D": "10", "E": "", "F": "15TH", "G": "ST", "H": "10 15TH ST, San Diego, CA", "I": "242", "J": "K", "K": "521", "L": "1", "M": "32.7057215", "N": "-117.1503498"},
{"A": "P17060008467", "B": "6/5/2017 19:27", "C": "2", "D": "10", "E": "", "F": "15TH", "G": "ST", "H": "10 15TH ST, San Diego, CA", "I": "5150", "J": "K", "K": "521", "L": "2", "M": "32.7057215", "N": "-117.1503498"},

我只想统计每种拨打电话的类型,拨打电话的时间,在哪个地点犯罪最多,在哪个日期犯罪最多等。

2 个答案:

答案 0 :(得分:0)

使用pandas

import pandas as pd

raw_df = pd.DataFrame(data)
df = raw_df.rename(columns=raw_df.iloc[0]).drop(0)
df

输出:

    incident_num        date_time day stno stdir1 StreetName      ...      call_type disposition beat priority         lat          long
1   P17060024503  6/14/2017 21:54   4   10              14TH      ...           1151           O  521        2  32.7054489  -117.1518696
2   P17030051227  3/29/2017 22:24   4   10              14TH      ...           1016           A  521        2  32.7054489  -117.1518696
3   P17060004814   6/3/2017 18:04   7   10              14TH      ...           1016           A  521        2  32.7054489  -117.1518696
4   P17030029336  3/17/2017 10:57   6   10              14TH      ...           1151          OT  521        2  32.7054489  -117.1518696
5   P17030005412   3/3/2017 23:45   6   10              15TH      ...           911P         CAN  521        2  32.7057215  -117.1503498
6   P17020016091   2/10/2017 8:23   6   10              15TH      ...            AU2           W  521        2  32.7057215  -117.1503498
7   P17040017368   4/11/2017 4:57   3   10              15TH      ...           5150         CAN  521        2  32.7057215  -117.1503498
8   P17030048050   3/28/2017 6:30   3   10              15TH      ...           1146           K  521           32.7057215  -117.1503498
9   P17060037341  6/22/2017 10:19   5   10              15TH      ...            242           K  521        1  32.7057215  -117.1503498
10  P17060008467   6/5/2017 19:27   2   10              15TH      ...           5150           K  521        2  32.7057215  -117.1503498

您可以运行的查询示例:

>>> df['call_type'].value_counts()
5150    2
1016    2
1151    2
242     1
911P    1
AU2     1
1146    1

答案 1 :(得分:0)

迭代json文件,并将所需字段存储在assosiatve数组中。您可以对其进行操作。

如果数据具有固定的列和结构,则可以将其存储在MySql之类的数据库中,并且可以通过简单的查询轻松地执行所需的操作。