我正在尝试将384.8mb的json文件导入Mongo,运行大约两分钟后发生错误导致无法导入。
warning: attempting to insert document with size 483.6 MB (exceeds 16.0 MB limit)
Mongo网站上的文档说bson的大小限制为16mb(http://docs.mongodb.org/manual/reference/limits/)。
导入此大小文件的推荐方法是什么?
以下是我的控制台的非详细输出
$ mongoimport -d myprovider -c collection --file data/myprovider_6am.json
2015-05-06T16:47:29.353-0700 connected to: localhost
2015-05-06T16:47:32.345-0700 [###############.........] myprovider.collection 256.0 MB/384.8 MB (66.5%)
2015-05-06T16:47:35.344-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:47:38.345-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:47:41.345-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:47:44.343-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:47:47.342-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:47:54.053-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:47:56.341-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:47:59.341-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:02.342-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:05.343-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:08.343-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:11.341-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:14.341-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:17.343-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:20.341-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:23.346-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:26.342-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:29.341-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:37.839-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:38.341-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:41.341-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:44.344-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:47.341-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:50.343-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:53.344-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:56.341-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:48:57.837-0700 warning: attempting to insert document with size 483.6 MB (exceeds 16.0 MB limit)
2015-05-06T16:48:59.341-0700 [########################] myprovider.collection 384.8 MB/384.8 MB (100.0%)
2015-05-06T16:49:00.306-0700 error inserting documents: write tcp 127.0.0.1:27017: broken pipe
2015-05-06T16:49:00.306-0700 imported 0 documents
答案 0 :(得分:1)
GridFS正是您要找的。 p>
如果您的文件大小超出16mb
,则可以存储在mongo的gridfs
中,该文件存储文件块而不是存储为单个文档。
除了存储大于16mb大小的文档之外,还有很多好处,比如在不将整个文件加载到内存中的情况下进行访问。
您还应该阅读When should I use GridFS?