Question

我需要使用JSON格式将数据写入文件。经过一番研究后，我尝试了以下代码

#!/usr/bin/perl -w

use JSON::XS;

my $count = 1;
my $nameExp;
my $numExp;
my $maps = [];

open my $out, '>', 'D:/Test.json';

my $json = JSON::XS->new->pretty(1)->utf8;

for (0..999999) {

   $numExp = "";
   for (0..9) {
      $numExp = $numExp.(int(rand(9)));
   }

   $nameExp = "";
   for (0..7) {
      $nameExp = $nameExp.(chr(int(rand(25) + 65)));
   }

   push @$maps, {ID => "$count", Name => "$nameExp", Number => "$numExp"};

   $count++;
}

print $out $json->encode({data => $maps});

我的问题是

如果键（地图内的列如ID，Name，Number等）或数据的数量（目前为1M）增加，则会产生在Out of memory!。
将数据推入阵列本身需要花费大量时间。然后我必须把它写到文件中。是否可以在保持正确格式的同时直接编写它？

我试图直接编写它而不是在数组中推送它，但格式无效。

我需要的样本JSON与下面的样本相同，

{
  "data": [
     {
        "ID": "1",
        "Name": "XXXXXX",
        "Number": "7670418426",
     },
     {
        "ID": "2",
        "Name": "YYYYYYY",
        "Number": "4421450424"
     }
  ]
}

任何建议或解决方案？

Answer 1

这有点像黑客，但你可以在循环中单独编码数组的每个元素，并立即将其写入磁盘。这需要单独打印页眉和页脚，并且必须手动引入元素之间的逗号

此程序创建一个较小的样本大小为10，但由于数据存储在磁盘而不是内存中，所以缩放没有问题

use strict;
use warnings;

use JSON::XS;

my $json_file = 'D:/Test.json';

open my $out, '>', $json_file
    or die qq{Unable to open "$json_file" for input: $!};
select $out;

print qq<{ "data": [\n>;

my $json = JSON::XS->new->pretty( 1 )->utf8;

my $count;
for ( 1 .. 10 ) {

    my $num_exp  = join '', map { int rand 10 } 1 .. 10;
    my $name_exp = join '', map { chr ord( 'A' ) + rand( 26 ) } 1 .. 8;

    print ', ' if $count;
    print $json->encode( { ID => ++$count, Name => $name_exp, Number => $num_exp } );
}

print qq<] }\n>;

输出

{ "data": [
{
   "Name" : "FQAJEWAL",
   "ID" : 1,
   "Number" : "6144230076"
}
, {
   "Number" : "4802605879",
   "Name" : "KKEBKUMZ",
   "ID" : 2
}
, {
   "Number" : "0843901915",
   "Name" : "SGBGIIAS",
   "ID" : 3
}
, {
   "Number" : "2255081597",
   "ID" : 4,
   "Name" : "ZJOPKMPP"
}
, {
   "Number" : "5392332416",
   "Name" : "DXHVGHGQ",
   "ID" : 5
}
, {
   "ID" : 6,
   "Name" : "HAYWEFWR",
   "Number" : "2727718733"
}
, {
   "ID" : 7,
   "Name" : "DIXEBUHW",
   "Number" : "9519451391"
}
, {
   "ID" : 8,
   "Name" : "LWIWIOTV",
   "Number" : "9892199187"
}
, {
   "ID" : 9,
   "Name" : "LJWFVEYC",
   "Number" : "4028143002"
}
, {
   "Name" : "ARNCHOXK",
   "ID" : 10,
   "Number" : "2550291006"
}
] }

更新

正确组装YAML数据要容易得多。这是一个生成YAML等效项的等效程序

use strict;
use warnings;

my $yaml_file = 'D:/Test.yaml';
open my $out, '>', $yaml_file
    or die qq{Unable to open "$yaml_file" for input: $!};
select $out;

print "---\n";
print "data:\n";

my $count;
for ( 1 .. 10 ) {

    my $num_exp  = join '', map { int rand 10 } 1 .. 10;
    my $name_exp = join '', map { chr ord( 'A' ) + rand( 26 ) } 1 .. 8;

    printf "- { ID: %d, Name: %s, Number: %s }\n", ++$count, $name_exp, $num_exp;
}

输出

---
data:
- { ID: 1, Name: UOCBQUTN, Number: 8349775867 }
- { ID: 2, Name: DXGMGTXH, Number: 3496217665 }
- { ID: 3, Name: JABBDZLQ, Number: 4207644646 }
- { ID: 4, Name: LWUJPOVY, Number: 5785602496 }
- { ID: 5, Name: JGYEGFDH, Number: 5056633664 }
- { ID: 6, Name: FUJKTBHF, Number: 6657427320 }
- { ID: 7, Name: NAKMJZRS, Number: 1318423549 }
- { ID: 8, Name: QXFDHPHP, Number: 1500434568 }
- { ID: 9, Name: BINACIAB, Number: 5759930882 }
- { ID: 10, Name: JSLNUXQE, Number: 9072575068 }

以JSON格式编写大量数据

1 个答案:

输出

更新

输出