Mongo导入,试图插入文件超过16mb的限制

时间:2015-05-07 00:04:22

标签: json mongodb

我正在尝试将384.8mb的json文件导入Mongo,运行大约两分钟后发生错误导致无法导入。

warning: attempting to insert document with size 483.6 MB (exceeds 16.0 MB limit)

Mongo网站上的文档说bson的大小限制为16mb(http://docs.mongodb.org/manual/reference/limits/)。

导入此大小文件的推荐方法是什么?

以下是我的控制台的非详细输出

$  mongoimport -d myprovider -c collection --file data/myprovider_6am.json
2015-05-06T16:47:29.353-0700  connected to: localhost
2015-05-06T16:47:32.345-0700  [###############.........] myprovider.collection  256.0 MB/384.8 MB (66.5%)
2015-05-06T16:47:35.344-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:47:38.345-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:47:41.345-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:47:44.343-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:47:47.342-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:47:54.053-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:47:56.341-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:47:59.341-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:02.342-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:05.343-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:08.343-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:11.341-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:14.341-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:17.343-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:20.341-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:23.346-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:26.342-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:29.341-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:37.839-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:38.341-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:41.341-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:44.344-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:47.341-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:50.343-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:53.344-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:56.341-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:57.837-0700  warning: attempting to insert document with size 483.6 MB (exceeds 16.0 MB limit)
2015-05-06T16:48:59.341-0700  [########################] myprovider.collection  384.8 MB/384.8 MB (100.0%)
2015-05-06T16:49:00.306-0700  error inserting documents: write tcp 127.0.0.1:27017: broken pipe
2015-05-06T16:49:00.306-0700  imported 0 documents

1 个答案:

答案 0 :(得分:1)

GridFS正是您要找的。

如果您的文件大小超出16mb,则可以存储在mongo的gridfs中,该文件存储文件块而不是存储为单个文档。

除了存储大于16mb大小的文档之外,还有很多好处,比如在不将整个文件加载到内存中的情况下进行访问。

您还应该阅读When should I use GridFS