Question

我正在尝试使用Java将大型JSON文件（newclicklogs.json）上传到mongodb。以下是我的JSON文件的样子：

{"preview":false,"result":{"search_term":"rania","request_time":"Sat Apr 01 12:47:04 -0400 2017","request_ip":"127.0.0.1","stats_type":"stats","upi":"355658761","unit":"DR","job_title":"Communications Officer","vpu":"INP","organization":"73","city":"Wash","country":"DC","title":"Tom","url":"www.demo.com","tab_name":"People-Tab","page_name":"PEOPLE","result_number":"5","page_num":"0","session_id":"df234f468cb3fe8be","total_results":"5","filter":"qterm=rina","_time":"2017-04-01T12:47:04.000-0400"}}
{"preview"......}
{"preview"......}
....

这是我的Java代码：

import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.commons.io.FileUtils;
import org.bson.Document;
import com.mongodb.DBObject;
import com.mongodb.MongoClient;

public class Main {

    public static void main(String[] args) throws IOException {

        String jsonString = FileUtils.readFileToString(new File("data/newclicklogs.json"), "UTF-8");

        Document doc = Document.parse(jsonString);
        List<Document> list = new ArrayList<>();
        list.add(doc);

        new MongoClient().getDatabase("test2").getCollection("collection1").insertMany(list);

    }
}

当我查询我的mongodb集合时，只添加了一个文档。如何将文件中的所有文档添加到mongodb集合中。我是mongodb的新手。任何帮助表示赞赏。

Answer 1

您应该尝试使用缓冲读取器进行批量写入。

以下代码将从文件中读取json数据，一次读取一行（文档），将json解析为Document并批量请求，然后再将其写入数据库。

MongoClient client = new MongoClient("localhost", 27017);
MongoDatabase database = client.getDatabase("test2");
MongoCollection<Document> collection = database.getCollection("collection1");

int count = 0;
int batch = 100;

List<InsertOneModel<Document>> docs = new ArrayList<>();

try (BufferedReader br = new BufferedReader(new FileReader("data/newclicklogs.json"))) {
      String line;
      while ((line = br.readLine()) != null) {
         docs.add(new InsertOneModel<>(Document.parse(line)));
         count++;
         if (count == batch) {
           collection.bulkWrite(docs, new BulkWriteOptions().ordered(false));
           docs.clear();
           count = 0;
        }
    }
}

if (count > 0) {
   collection.bulkWrite(docs, new BulkWriteOptions().ordered(false));
}

当您在整个json上运行Document.parse时，您实际上是通过覆盖以前的所有文档来将文档缩减到最后一个文档。

更多信息

http://mongodb.github.io/mongo-java-driver/3.4/driver/tutorials/bulk-writes/

如何使用Java上传mongodb中的json文件？

1 个答案: