MongoDB mongoimport在导入大型json文件时出错

时间:2014-10-05 17:07:21

标签: json mongodb import

我有一个大型的JSON格式文件(254MB),我试图使用mongoimport导入Cloud9 IDE中的MongoDB。 Json文件由mongodb文档组成,每个文档都使用换行符。该文件如下所示:

...
{"t":"1358836264","p":"1.33470"}
{"t":"1358836265","p":"1.33475"}
{"t":"1358836271","p":"1.33477"}
{"t":"1358836272","p":"1.33481"}
{"t":"1358836274","p":"1.33475"}
{"t":"1358836275","p":"1.33478"}
{"t":"1358836288","p":"1.33480"}
{"t":"1358836291","p":"1.33481"}
{"t":"1358836294","p":"1.33481"}
{"t":"1358836295","p":"1.33478"}
...

我试过了:

mongoimport -d woption -c eurusd eurusddata.json

这给了我以下输出错误:

connected to: 127.0.0.1
Sun Oct  5 16:51:15.094                 Progress: 1636800/26345472      6%
Sun Oct  5 16:51:15.094                         49600   16533/second
Sun Oct  5 16:51:18.088                 Progress: 3375900/26345472      12%
Sun Oct  5 16:51:18.088                         102300  17050/second
Sun Oct  5 16:51:21.089                 Progress: 4867500/26345472      18%
Sun Oct  5 16:51:21.089                         147500  16388/second
Sun Oct  5 16:51:24.103                 Progress: 7728600/26345472      29%
Sun Oct  5 16:51:24.103                         234200  19516/second
Sun Oct  5 16:51:27.093                 Progress: 10467600/26345472     39%
Sun Oct  5 16:51:27.093                         317200  21146/second
Sun Oct  5 16:51:30.094                 Progress: 13312200/26345472     50%
Sun Oct  5 16:51:30.094                         403400  22411/second
Sun Oct  5 16:51:33.302                 Progress: 16038000/26345472     60%
Sun Oct  5 16:51:33.303                         486000  23142/second
Sun Oct  5 16:51:36.088                 Progress: 17341500/26345472     65%
Sun Oct  5 16:51:36.088                         525500  21895/second
Sun Oct  5 16:51:39.004                 Progress: 18526200/26345472     70%
Sun Oct  5 16:51:39.004                         561400  20792/second
Sun Oct  5 16:51:42.032                 Progress: 19067400/26345472     72%
Sun Oct  5 16:51:42.032                         577800  19260/second
Sun Oct  5 16:51:45.088                 Progress: 20829600/26345472     79%
Sun Oct  5 16:51:45.088                         631200  19127/second
Sun Oct  5 16:51:48.071                 Progress: 23007600/26345472     87%
Sun Oct  5 16:51:48.071                         697200  19366/second
Sun Oct  5 16:51:51.914                 Progress: 23443200/26345472     88%
Sun Oct  5 16:51:51.914                         710400  18215/second
Sun Oct  5 16:51:54.103                 Progress: 23611500/26345472     89%
Sun Oct  5 16:51:54.104                         715500  17035/second
Sun Oct  5 16:51:57.709                 Progress: 23967900/26345472     90%
Sun Oct  5 16:51:57.709                         726300  16140/second
Sun Oct  5 16:52:00.096                 Progress: 24538800/26345472     93%
Sun Oct  5 16:52:00.096                         743600  15491/second
Sun Oct  5 16:52:03.088                 Progress: 25548600/26345472     96%
Sun Oct  5 16:52:03.088                         774200  15180/second
Sun Oct  5 16:52:04.644 exception:BSON representation of supplied JSON is too large: code FailedToParse: FailedToParse: Expecting ':': offset:21
Sun Oct  5 16:52:04.644 
Sun Oct  5 16:52:04.644 check 9 798347
Sun Oct  5 16:52:05.061 imported 798347 objects
Sun Oct  5 16:52:05.061 ERROR: encountered 1 error(s)

然后我尝试在上一个命令结束时使用--jsonArray:

mongoimport -d woption -c eurusd eurusddata.json --jsonArray

这给了我以下输出错误:

connected to: 127.0.0.1
Sun Oct  5 16:54:20.343 exception:JSONArray file too large
Sun Oct  5 16:54:20.444 warning: log line attempted (16384k) over max size(10k), printing beginning and end ... {"t":"1357070402","p":"1.32041"}
{"t":"1357070424","p":"1.32040"}
{"t":"1357070447","p":"1.32038"}
{"t":"1357070457","p":"1.32034"}
{"t":"1357070463","p":"1.32039"}
{"t":"1357070464","p":"1.32038"}
{"t":"1357070470","p":"1.32034"}
{"t":"1357070485","p":"1.32035"}
{"t":"1357070491","p":"1.32046"}
{"t":"1357070520","p":"1.32050"}
{"t":"1357070522","p":"1.32049"}
{"t":"1357070535","p":"1.32050"}
{"t":"1357070581","p":"1.32049"}
{"t":"1357070582","p":"1.32050"}
{"t":"1357070587","p":"1.32035"}
{"t":"1357070589","p":"1.32034"}
{"t":"1357070593","p":"1.32033"}
{"t":"1357070594","p":"1.32024"}
{"t":"1357070595","p":"1.32025"}
{"t":"1357070599","p":"1.32009"}
{"t":"1357070602","p":"1.32024"}
{"t":"1357070630","p":"1.32023"}
{"t":"1357070637","p":"1.32025"}
{"t":"1357070656","p":"1.32023"}
{"t":"1357070691","p":"1.32025"}
{"t":"1357070702","p":"1.32027"}
{"t":"1357070703","p":"1.32028"}
{"t":"1357070706","p":"1.32027"}
{"t":"1357070707","p":"1.32026"}
{"t":"1357070709","p":"1.32027"}
{"t":"1357070710","p":"1.32026"}
{"t":"1357070721","p":"1.32027"}
{"t":"1357070723","p":"1.32026"}
{"t":"1357070761","p":"1.32025"}
{"t":"1357070767","p":"1.32027"}
{"t":"1357070768","p":"1.32025"}
{"t":"1357070784","p":"1.32026"}
{"t":"1357070798","p":"1.32027"}
{"t":"1357070799","p":"1.32026"}
{"t":"1357070804","p":"1.32036"}
{"t":"1357070819","p":"1.32034"}
{"t":"1357070842","p":"1.32035"}
{"t":"1357070880","p":"1.32030"}
{"t":"1357070881","p":"1.32035"}
{"t":"1357070966","p":"1.32053"}
{"t":"1357070967","p":"1.32063"}
{"t":"1357070973","p":"1.32048"}
{"t":"1357070974","p":"1.32046"}
{"t":"1357070977","p":"1.32066"}
{"t":"1357070978","p":"1.32065"}
{"t":"1357070984","p":"1.32064"}
{"t":"1357070987","p":"1.32063"}
{"t":"1357070988","p":"1.32064"}
{"t":"1357070993","p":"1.32065"}
{"t":"1357070997","p":"1.32065"}
{"t":"1357070998","p":"1.32063"}
{"t":"1357071000","p":"1.32064"}
{"t":"1357071003","p":"1.32064"}
{"t":"1357071011","p":"1.32065"}
{"t":"1357071012","p":"1.32069"}
{"t":"1357071013","p":"1.32069"}
{"t":"1357071023","p":"1.32068"}
{"t":"1357071024","p":"1.32068"}
{"t":"1357071026","p":"1.32072"}
{"t":"1357071027","p":"1.32070"}
{"t":"1357071028","p":"1.32069"}
{"t":"1357071047","p":"1.32068"}
{"t":"1357071048","p":"1.32069"}
{"t":"1357071070","p":"1.32072"}
{"t":"1357071071","p":"1.32069"}
{"t":"1357071077","p":"1.32068"}
{"t":"1357071078","p":"1.32069"}
{"t":"1357071085","p":"1.32068"}
{"t":"1357071086","p":"1.32069"}
{"t":"1357071095","p":"1.32072"}
{"t":"1357071096","p":"1.32069"}
{"t":"1357071097","p":"1.32070"}
{"t":"1357071104","p":"1.32069"}
{"t":"1357071105","p":"1.32069"}
{"t":"1357071114","p":"1.32068"}
{"t":"1357071125","p":"1.32068"}
{"t":"1357071132","p":"1.32072"}
{"t":"1357071133","p":"1.32068"}
{"t":"1357071134","p":"1.32069"}
{"t":"1357071136","p":"1.32043"}
{"t":"1357071137","p":"1.32049"}
{"t":"1357071139","p":"1.32052"}
{"t":"1357071140","p":"1.32058"}
{"t":"1357071156","p":"1.32052"}
{"t":"1357071163","p":"1.32050"}
{"t":"1357071174","p":"1.32052"}
{"t":"1357071176","p":"1.32055"}
{"t":"1357071178","p":"1.32053"}
{"t":"1357071185","p":"1.32054"}
{"t":"1357071187","p":"1.32053"}
{"t":"1357071189","p":"1.32025"}
{"t":"1357071190","p":"1.32023"}
{"t":"1357071191","p":"1.32045"}
{"t":"1357071192","p":"1.32044"}
{"t":"1357071193","p":"1.32045"}
{"t":"1357071197","p":"1.32044"}
{"t":"1357071198","p":"1.32043"}
{"t":"1357071199","p":"1.32004"}
{"t":"13570712 .......... ":"1358836127","p":"1.33455"}
{"t":"1358836128","p":"1.33454"}
{"t":"1358836129","p":"1.33455"}
{"t":"1358836130","p":"1.33458"}
{"t":"1358836131","p":"1.33463"}
{"t":"1358836132","p":"1.33462"}
{"t":"1358836133","p":"1.33460"}
{"t":"1358836134","p":"1.33459"}
{"t":"1358836135","p":"1.33464"}
{"t":"1358836137","p":"1.33463"}
{"t":"1358836139","p":"1.33461"}
{"t":"1358836140","p":"1.33462"}
{"t":"1358836142","p":"1.33463"}
{"t":"1358836143","p":"1.33459"}
{"t":"1358836144","p":"1.33461"}
{"t":"1358836146","p":"1.33462"}
{"t":"1358836147","p":"1.33466"}
{"t":"1358836148","p":"1.33473"}
{"t":"1358836149","p":"1.33475"}
{"t":"1358836151","p":"1.33479"}
{"t":"1358836152","p":"1.33484"}
{"t":"1358836153","p":"1.33492"}
{"t":"1358836154","p":"1.33488"}
{"t":"1358836155","p":"1.33492"}
{"t":"1358836156","p":"1.33489"}
{"t":"1358836157","p":"1.33482"}
{"t":"1358836158","p":"1.33483"}
{"t":"1358836159","p":"1.33480"}
{"t":"1358836160","p":"1.33479"}
{"t":"1358836161","p":"1.33482"}
{"t":"1358836162","p":"1.33481"}
{"t":"1358836163","p":"1.33481"}
{"t":"1358836166","p":"1.33480"}
{"t":"1358836167","p":"1.33480"}
{"t":"1358836168","p":"1.33475"}
{"t":"1358836169","p":"1.33454"}
{"t":"1358836170","p":"1.33454"}
{"t":"1358836172","p":"1.33459"}
{"t":"1358836173","p":"1.33456"}
{"t":"1358836174","p":"1.33455"}
{"t":"1358836175","p":"1.33457"}
{"t":"1358836176","p":"1.33459"}
{"t":"1358836177","p":"1.33460"}
{"t":"1358836178","p":"1.33462"}
{"t":"1358836179","p":"1.33458"}
{"t":"1358836180","p":"1.33459"}
{"t":"1358836181","p":"1.33456"}
{"t":"1358836182","p":"1.33458"}
{"t":"1358836183","p":"1.33457"}
{"t":"1358836184","p":"1.33459"}
{"t":"1358836185","p":"1.33460"}
{"t":"1358836186","p":"1.33467"}
{"t":"1358836189","p":"1.33466"}
{"t":"1358836190","p":"1.33472"}
{"t":"1358836191","p":"1.33470"}
{"t":"1358836192","p":"1.33470"}
{"t":"1358836193","p":"1.33469"}
{"t":"1358836194","p":"1.33465"}
{"t":"1358836195","p":"1.33466"}
{"t":"1358836196","p":"1.33461"}
{"t":"1358836197","p":"1.33460"}
{"t":"1358836198","p":"1.33466"}
{"t":"1358836199","p":"1.33465"}
{"t":"1358836200","p":"1.33466"}
{"t":"1358836201","p":"1.33458"}
{"t":"1358836202","p":"1.33457"}
{"t":"1358836203","p":"1.33453"}
{"t":"1358836204","p":"1.33454"}
{"t":"1358836205","p":"1.33456"}
{"t":"1358836210","p":"1.33455"}
{"t":"1358836211","p":"1.33450"}
{"t":"1358836214","p":"1.33449"}
{"t":"1358836216","p":"1.33451"}
{"t":"1358836217","p":"1.33452"}
{"t":"1358836218","p":"1.33457"}
{"t":"1358836220","p":"1.33458"}
{"t":"1358836221","p":"1.33457"}
{"t":"1358836222","p":"1.33456"}
{"t":"1358836223","p":"1.33457"}
{"t":"1358836226","p":"1.33460"}
{"t":"1358836229","p":"1.33459"}
{"t":"1358836230","p":"1.33458"}
{"t":"1358836236","p":"1.33461"}
{"t":"1358836237","p":"1.33466"}
{"t":"1358836240","p":"1.33467"}
{"t":"1358836241","p":"1.33467"}
{"t":"1358836242","p":"1.33463"}
{"t":"1358836243","p":"1.33466"}
{"t":"1358836244","p":"1.33467"}
{"t":"1358836245","p":"1.33468"}
{"t":"1358836250","p":"1.33468"}
{"t":"1358836256","p":"1.33469"}
{"t":"1358836263","p":"1.33472"}
{"t":"1358836264","p":"1.33470"}
{"t":"1358836265","p":"1.33475"}
{"t":"1358836271","p":"1.33477"}
{"t":"1358836272","p":"1.33481"}
{"t":"1358836274","p":"1.33475"}
{"t":"1358836275","p":"1.33478"}
{"t":"1358836288","p":"1.33480"}
{"t":"1358836291","p":"1.33481"}
{"t":"1358836294","p":"1.33481"}
{"t":"1358836295","p":"1.33478"}
{"t":"1358836296
Sun Oct  5 16:54:20.445 check 0 0
Sun Oct  5 16:54:20.445 imported 0 objects
Sun Oct  5 16:54:20.445 ERROR: encountered 1 error(s)

然后我尝试使用这样的东西:

mongoimport -d woption -c eurusd < /home/ubuntu/workspace/eurusddata.json

这给了我以下输出错误:

connected to: 127.0.0.1
Sun Oct  5 16:58:08.107                         77400   25800/second
Sun Oct  5 16:58:11.096                         136300  22716/second
Sun Oct  5 16:58:14.107                         177700  19744/second
Sun Oct  5 16:58:17.093                         232900  19408/second
Sun Oct  5 16:58:20.125                         280400  18693/second
Sun Oct  5 16:58:23.103                         343500  19083/second
Sun Oct  5 16:58:26.091                         424800  20228/second
Sun Oct  5 16:58:29.095                         482800  20116/second
Sun Oct  5 16:58:32.097                         517900  19181/second
Sun Oct  5 16:58:35.088                         566700  18890/second
Sun Oct  5 16:58:38.091                         608700  18445/second
Sun Oct  5 16:58:41.098                         664700  18463/second
Sun Oct  5 16:58:44.088                         718300  18417/second
Sun Oct  5 16:58:47.089                         776800  18495/second
Sun Oct  5 16:58:47.848 exception:BSON representation of supplied JSON is too large: code FailedToParse: FailedToParse: Expecting ':': offset:21
Sun Oct  5 16:58:47.848 
Sun Oct  5 16:58:47.848 check 9 798347
Sun Oct  5 16:58:48.852 imported 798347 objects
Sun Oct  5 16:58:48.852 ERROR: encountered 1 error(s)

还尝试使用以前的命令--jsonArray:

mongoimport -d woption -c eurusd < /home/ubuntu/workspace/eurusddata.json --jsonArray

这给了我以下输出错误:

connected to: 127.0.0.1
Sun Oct  5 16:59:35.063 exception:JSONArray file too large
Sun Oct  5 16:59:35.101 warning: log line attempted (16384k) over max size(10k), printing beginning and end ... {"t":"1357070402","p":"1.32041"}
{"t":"1357070424","p":"1.32040"}
{"t":"1357070447","p":"1.32038"}
{"t":"1357070457","p":"1.32034"}
{"t":"1357070463","p":"1.32039"}
{"t":"1357070464","p":"1.32038"}
{"t":"1357070470","p":"1.32034"}
{"t":"1357070485","p":"1.32035"}
{"t":"1357070491","p":"1.32046"}
{"t":"1357070520","p":"1.32050"}
{"t":"1357070522","p":"1.32049"}
{"t":"1357070535","p":"1.32050"}
{"t":"1357070581","p":"1.32049"}
{"t":"1357070582","p":"1.32050"}
{"t":"1357070587","p":"1.32035"}
{"t":"1357070589","p":"1.32034"}
{"t":"1357070593","p":"1.32033"}
{"t":"1357070594","p":"1.32024"}
{"t":"1357070595","p":"1.32025"}
{"t":"1357070599","p":"1.32009"}
{"t":"1357070602","p":"1.32024"}
{"t":"1357070630","p":"1.32023"}
{"t":"1357070637","p":"1.32025"}
{"t":"1357070656","p":"1.32023"}
{"t":"1357070691","p":"1.32025"}
{"t":"1357070702","p":"1.32027"}
{"t":"1357070703","p":"1.32028"}
{"t":"1357070706","p":"1.32027"}
{"t":"1357070707","p":"1.32026"}
{"t":"1357070709","p":"1.32027"}
{"t":"1357070710","p":"1.32026"}
{"t":"1357070721","p":"1.32027"}
{"t":"1357070723","p":"1.32026"}
{"t":"1357070761","p":"1.32025"}
{"t":"1357070767","p":"1.32027"}
{"t":"1357070768","p":"1.32025"}
{"t":"1357070784","p":"1.32026"}
{"t":"1357070798","p":"1.32027"}
{"t":"1357070799","p":"1.32026"}
{"t":"1357070804","p":"1.32036"}
{"t":"1357070819","p":"1.32034"}
{"t":"1357070842","p":"1.32035"}
{"t":"1357070880","p":"1.32030"}
{"t":"1357070881","p":"1.32035"}
{"t":"1357070966","p":"1.32053"}
{"t":"1357070967","p":"1.32063"}
{"t":"1357070973","p":"1.32048"}
{"t":"1357070974","p":"1.32046"}
{"t":"1357070977","p":"1.32066"}
{"t":"1357070978","p":"1.32065"}
{"t":"1357070984","p":"1.32064"}
{"t":"1357070987","p":"1.32063"}
{"t":"1357070988","p":"1.32064"}
{"t":"1357070993","p":"1.32065"}
{"t":"1357070997","p":"1.32065"}
{"t":"1357070998","p":"1.32063"}
{"t":"1357071000","p":"1.32064"}
{"t":"1357071003","p":"1.32064"}
{"t":"1357071011","p":"1.32065"}
{"t":"1357071012","p":"1.32069"}
{"t":"1357071013","p":"1.32069"}
{"t":"1357071023","p":"1.32068"}
{"t":"1357071024","p":"1.32068"}
{"t":"1357071026","p":"1.32072"}
{"t":"1357071027","p":"1.32070"}
{"t":"1357071028","p":"1.32069"}
{"t":"1357071047","p":"1.32068"}
{"t":"1357071048","p":"1.32069"}
{"t":"1357071070","p":"1.32072"}
{"t":"1357071071","p":"1.32069"}
{"t":"1357071077","p":"1.32068"}
{"t":"1357071078","p":"1.32069"}
{"t":"1357071085","p":"1.32068"}
{"t":"1357071086","p":"1.32069"}
{"t":"1357071095","p":"1.32072"}
{"t":"1357071096","p":"1.32069"}
{"t":"1357071097","p":"1.32070"}
{"t":"1357071104","p":"1.32069"}
{"t":"1357071105","p":"1.32069"}
{"t":"1357071114","p":"1.32068"}
{"t":"1357071125","p":"1.32068"}
{"t":"1357071132","p":"1.32072"}
{"t":"1357071133","p":"1.32068"}
{"t":"1357071134","p":"1.32069"}
{"t":"1357071136","p":"1.32043"}
{"t":"1357071137","p":"1.32049"}
{"t":"1357071139","p":"1.32052"}
{"t":"1357071140","p":"1.32058"}
{"t":"1357071156","p":"1.32052"}
{"t":"1357071163","p":"1.32050"}
{"t":"1357071174","p":"1.32052"}
{"t":"1357071176","p":"1.32055"}
{"t":"1357071178","p":"1.32053"}
{"t":"1357071185","p":"1.32054"}
{"t":"1357071187","p":"1.32053"}
{"t":"1357071189","p":"1.32025"}
{"t":"1357071190","p":"1.32023"}
{"t":"1357071191","p":"1.32045"}
{"t":"1357071192","p":"1.32044"}
{"t":"1357071193","p":"1.32045"}
{"t":"1357071197","p":"1.32044"}
{"t":"1357071198","p":"1.32043"}
{"t":"1357071199","p":"1.32004"}
{"t":"13570712 .......... ":"1358836127","p":"1.33455"}
{"t":"1358836128","p":"1.33454"}
{"t":"1358836129","p":"1.33455"}
{"t":"1358836130","p":"1.33458"}
{"t":"1358836131","p":"1.33463"}
{"t":"1358836132","p":"1.33462"}
{"t":"1358836133","p":"1.33460"}
{"t":"1358836134","p":"1.33459"}
{"t":"1358836135","p":"1.33464"}
{"t":"1358836137","p":"1.33463"}
{"t":"1358836139","p":"1.33461"}
{"t":"1358836140","p":"1.33462"}
{"t":"1358836142","p":"1.33463"}
{"t":"1358836143","p":"1.33459"}
{"t":"1358836144","p":"1.33461"}
{"t":"1358836146","p":"1.33462"}
{"t":"1358836147","p":"1.33466"}
{"t":"1358836148","p":"1.33473"}
{"t":"1358836149","p":"1.33475"}
{"t":"1358836151","p":"1.33479"}
{"t":"1358836152","p":"1.33484"}
{"t":"1358836153","p":"1.33492"}
{"t":"1358836154","p":"1.33488"}
{"t":"1358836155","p":"1.33492"}
{"t":"1358836156","p":"1.33489"}
{"t":"1358836157","p":"1.33482"}
{"t":"1358836158","p":"1.33483"}
{"t":"1358836159","p":"1.33480"}
{"t":"1358836160","p":"1.33479"}
{"t":"1358836161","p":"1.33482"}
{"t":"1358836162","p":"1.33481"}
{"t":"1358836163","p":"1.33481"}
{"t":"1358836166","p":"1.33480"}
{"t":"1358836167","p":"1.33480"}
{"t":"1358836168","p":"1.33475"}
{"t":"1358836169","p":"1.33454"}
{"t":"1358836170","p":"1.33454"}
{"t":"1358836172","p":"1.33459"}
{"t":"1358836173","p":"1.33456"}
{"t":"1358836174","p":"1.33455"}
{"t":"1358836175","p":"1.33457"}
{"t":"1358836176","p":"1.33459"}
{"t":"1358836177","p":"1.33460"}
{"t":"1358836178","p":"1.33462"}
{"t":"1358836179","p":"1.33458"}
{"t":"1358836180","p":"1.33459"}
{"t":"1358836181","p":"1.33456"}
{"t":"1358836182","p":"1.33458"}
{"t":"1358836183","p":"1.33457"}
{"t":"1358836184","p":"1.33459"}
{"t":"1358836185","p":"1.33460"}
{"t":"1358836186","p":"1.33467"}
{"t":"1358836189","p":"1.33466"}
{"t":"1358836190","p":"1.33472"}
{"t":"1358836191","p":"1.33470"}
{"t":"1358836192","p":"1.33470"}
{"t":"1358836193","p":"1.33469"}
{"t":"1358836194","p":"1.33465"}
{"t":"1358836195","p":"1.33466"}
{"t":"1358836196","p":"1.33461"}
{"t":"1358836197","p":"1.33460"}
{"t":"1358836198","p":"1.33466"}
{"t":"1358836199","p":"1.33465"}
{"t":"1358836200","p":"1.33466"}
{"t":"1358836201","p":"1.33458"}
{"t":"1358836202","p":"1.33457"}
{"t":"1358836203","p":"1.33453"}
{"t":"1358836204","p":"1.33454"}
{"t":"1358836205","p":"1.33456"}
{"t":"1358836210","p":"1.33455"}
{"t":"1358836211","p":"1.33450"}
{"t":"1358836214","p":"1.33449"}
{"t":"1358836216","p":"1.33451"}
{"t":"1358836217","p":"1.33452"}
{"t":"1358836218","p":"1.33457"}
{"t":"1358836220","p":"1.33458"}
{"t":"1358836221","p":"1.33457"}
{"t":"1358836222","p":"1.33456"}
{"t":"1358836223","p":"1.33457"}
{"t":"1358836226","p":"1.33460"}
{"t":"1358836229","p":"1.33459"}
{"t":"1358836230","p":"1.33458"}
{"t":"1358836236","p":"1.33461"}
{"t":"1358836237","p":"1.33466"}
{"t":"1358836240","p":"1.33467"}
{"t":"1358836241","p":"1.33467"}
{"t":"1358836242","p":"1.33463"}
{"t":"1358836243","p":"1.33466"}
{"t":"1358836244","p":"1.33467"}
{"t":"1358836245","p":"1.33468"}
{"t":"1358836250","p":"1.33468"}
{"t":"1358836256","p":"1.33469"}
{"t":"1358836263","p":"1.33472"}
{"t":"1358836264","p":"1.33470"}
{"t":"1358836265","p":"1.33475"}
{"t":"1358836271","p":"1.33477"}
{"t":"1358836272","p":"1.33481"}
{"t":"1358836274","p":"1.33475"}
{"t":"1358836275","p":"1.33478"}
{"t":"1358836288","p":"1.33480"}
{"t":"1358836291","p":"1.33481"}
{"t":"1358836294","p":"1.33481"}
{"t":"1358836295","p":"1.33478"}
{"t":"1358836296
Sun Oct  5 16:59:35.101 check 0 0
Sun Oct  5 16:59:35.102 imported 0 objects
Sun Oct  5 16:59:35.102 ERROR: encountered 1 error(s)

我做错了什么或我在这里错过了什么?如何将大型json文件导入MongoDB?

3 个答案:

答案 0 :(得分:2)

根据消息:

Sun Oct  5 16:52:03.088                 Progress: 25548600/26345472     96%
Sun Oct  5 16:52:03.088                         774200  15180/second
Sun Oct  5 16:52:04.644 exception:BSON representation of supplied JSON is too large: code FailedToParse: FailedToParse: Expecting ':': offset:21  

我认为导入文件末尾附近会出现一些错误。

对于{"t":"1358836291","p":"1.33481"},第二个:的偏移量为21.因此,请检查导入文件末尾(96%~100%)附近的文档语法。
顺便说一下,--jsonArray在这里使用是不合适的,因为它有16MB的限制。

答案 1 :(得分:0)

将您的命令从mongoimport -d woption -c eurusd eurusddata.json --jsonArray更改为mongoimport -d woption -c eurusd --file eurusddata.json --jsonArray,这将有效。希望这可以帮助。如果这不起作用,请更新。

答案 2 :(得分:0)

尝试此选项: - batchSize 1

喜欢:

    mongoimport -d woption -c eurusd eurusddata.json --batchSize 1