我已经获得了一个应用程序,其中输入已从50K位置记录扩展到110万个位置记录。 这导致了严重的问题,因为整个文件先前被反序列化为单个对象。 对于具有110万条记录的生产类文件,对象的大小约为1GB。 由于大对象GC问题,我希望将反序列化对象保持在85K以下。
我试图一次解析一个位置对象并对其进行反序列化,以便控制对象的数量 得到反序列化,然后控制对象的大小。我使用Json.Net库来做到这一点。
以下是我作为流进入我的应用程序的JSON文件示例。
{
"Locations": [{
"LocationId": "",
"ParentLocationId": "",
"DisplayFlag": "Y",
"DisplayOptions": "",
"DisplayName": "",
"Address": "",
"SecondaryAddress": "",
"City": "",
"State": "",
"PostalCode": "",
"Country": "",
"Latitude": 40.59485,
"Longitude": -73.96174,
"LatLonQuality": 99,
"BusinessLogoUrl": "",
"BusinessUrl": "",
"DisplayText": "",
"PhoneNumber": "",
"VenueGroup": 7,
"VenueType": 0,
"SubVenue": 0,
"IndoorFlag": "",
"OperatorDefined": "",
"AccessPoints": [{
"AccessPointId": "",
"MACAddress": "",
"DisplayFlag": "",
"DisplayOptions": "",
"Latitude": 40.59485,
"Longitude": -73.96174,
"Status": "Up",
"OperatorDefined": "",
"RoamingGroups": [{
"GroupName": ""
},
{
"GroupName": ""
}],
"Radios": [{
"RadioId": "",
"RadioFrequency": "",
"RadioProtocols": [{
"Protocol": ""
}],
"WifiConnections": [{
"BSSID": "",
"ServiceSets": [{
"SSID": "",
"SSID_Broadcasted": ""
}]
}]
}]
}]
},
{
"LocationId": "",
"ParentLocationId": "",
"DisplayFlag": "Y",
"DisplayOptions": "",
"DisplayName": "",
"Address": "",
"SecondaryAddress": "",
"City": "",
"State": "",
"PostalCode": "",
"Country": "",
"Latitude": 40.59485,
"Longitude": -73.96174,
"LatLonQuality": 99,
"BusinessLogoUrl": "",
"BusinessUrl": "",
"DisplayText": "",
"PhoneNumber": "",
"VenueGroup": 7,
"VenueType": 0,
"SubVenue": 0,
"IndoorFlag": "",
"OperatorDefined": "",
"AccessPoints": [{
"AccessPointId": "",
"MACAddress": "",
"DisplayFlag": "",
"DisplayOptions": "",
"Latitude": 40.59485,
"Longitude": -73.96174,
"Status": "Up",
"OperatorDefined": "",
"RoamingGroups": [{
"GroupName": ""
},
{
"GroupName": ""
}],
"Radios": [{
"RadioId": "",
"RadioFrequency": "",
"RadioProtocols": [{
"Protocol": ""
}],
"WifiConnections": [{
"BSSID": "",
"ServiceSets": [{
"SSID": "",
"SSID_Broadcasted": ""
}]
}]
}]
}]
}]
}
我需要能够拉出各个Location对象,以便我查看以下内容
{
"LocationId": "",
"ParentLocationId": "",
"DisplayFlag": "Y",
"DisplayOptions": "",
"DisplayName": "",
"Address": "",
"SecondaryAddress": "",
"City": "",
"State": "",
"PostalCode": "",
"Country": "",
"Latitude": 40.59485,
"Longitude": -73.96174,
"LatLonQuality": 99,
"BusinessLogoUrl": "",
"BusinessUrl": "",
"DisplayText": "",
"PhoneNumber": "",
"VenueGroup": 7,
"VenueType": 0,
"SubVenue": 0,
"IndoorFlag": "",
"OperatorDefined": "",
"AccessPoints": [{
"AccessPointId": "",
"MACAddress": "",
"DisplayFlag": "",
"DisplayOptions": "",
"Latitude": 40.59485,
"Longitude": -73.96174,
"Status": "Up",
"OperatorDefined": "",
"RoamingGroups": [{
"GroupName": ""
},
{
"GroupName": ""
}],
"Radios": [{
"RadioId": "",
"RadioFrequency": "",
"RadioProtocols": [{
"Protocol": ""
}],
"WifiConnections": [{
"BSSID": "",
"ServiceSets": [{
"SSID": "",
"SSID_Broadcasted": ""
}]
}]
}]
}]
}
我试图使用Json.NET JsonTextReader来实现这一点,但是我无法让读者在其缓冲区中包含整个位置,因为读者最初将拥有的流中的记录大小直到" RadioProtocols",这是通过对象的中途,当流到达对象的末尾时,读者已经放弃了对象的开始。
我尝试使用此功能的代码是
var ser = new JsonSerializer();
using (var reader = new JsonTextReader(new StreamReader(stream)))
{
reader.SupportMultipleContent = true;
while (reader.Read())
{
if (reader.TokenType == JsonToken.StartObject && reader.Depth == 2)
{
do
{
reader.Read();
} while (reader.TokenType != JsonToken.EndObject && reader.Depth == 2);
var singleLocation = ser.Deserialize<Locations>(reader);
}
}
}
有关此的任何信息或替代方法将非常感谢。作为旁注,我们的客户发送信息的方式目前无法改变。
答案 0 :(得分:0)
当读者位于要反序列化的对象的开头(在您的情况下为Locations
数组中的条目)时,您只需调用ser.Deserialize<T>(reader)
它就会起作用,前进到在该级别的对象的结束,没有进一步。因此,以下内容应迭代文件中的Location
个对象,分别加载每个对象:
public static IEnumerable<T> DeserializeNestedItems<T>(TextReader textReader)
{
var ser = new JsonSerializer();
using (var reader = new JsonTextReader(textReader))
{
reader.SupportMultipleContent = true;
while (reader.Read())
{
if (reader.TokenType == JsonToken.StartObject && reader.Depth == 2)
{
var item = ser.Deserialize<T>(reader);
yield return item;
}
}
}
}
使用测试字符串的一个使用示例:
Debug.Assert(DeserializeNestedItems<Location>(new StringReader(json)).Count() == 2); // No assert.
var list = DeserializeNestedItems<Location>(new StringReader(json)).SelectMany(l => l.AccessPoints).Select(a => new { a.Latitude, a.Longitude }).ToList();
Debug.WriteLine(JsonConvert.SerializeObject(list, Formatting.Indented));
哪个输出:
[ { "Latitude": 40.59485, "Longitude": -73.96174 }, { "Latitude": 40.59485, "Longitude": -73.96174 } ]
注意 - Location
课程来自将您的JSON发布到http://json2csharp.com/。
答案 1 :(得分:0)
感谢所有的帮助,我设法让它按照我想要的方式进行反序列化单个位置对象。
如果项目被转换为JObject,它将读入完整对象并对其进行反序列化,这可以循环以获得解决方案。
这是在
上解决的代码while (reader.Read())
{
if (reader.TokenType == JsonToken.StartObject && reader.Depth == 2)
{
location = JObject.Load(reader).ToObject<Location>();
var lv = new LocationValidator(location, FootprintInfo.OperatorId, FootprintInfo.RoamingGroups, true);
var vr = lv.IsValid();
if (vr.Successful)
{
yield return location;
}
else
{
errors.Add(new Error(elNumber, location.LocationId, vr.Error.Field, vr.Error.Detail));
if (errors.Count >= maxErrors)
{
yield break;
}
}
++elNumber;
}
}