Question

根据官方MongoDB documentation，自3.6版以来，批量插入的限制为100,000个操作（相比之下，旧版本中为1000个操作）。

作为一个基准，使用Java客户端，我将100K文档以1到100K的不同批量插入到集合中。这些是一些配置的结果：

我看到3.2版（与3.6版一样）对于大于500-1000 ops的批量产品提供了相同的结果。

有关最后一个基准测试设置的一些信息：

6个虚拟CPU
Windows Server 2012
128GB RAM
运行mongod的MongoDB 3.6版
wiredTigerCacheSizeGB = 20
插入的文档具有5个字段：fName，lName，text1，text2和text3。后三个字段包含“ Lorem ipsum”的相似段落。
Java mongoDB客户端：org.mongodb.mongo-java-driver版本3.8.0

这是基准代码：

    public static void main(String[] args) {


        MongoCredential credential = MongoCredential.createCredential("user", "admin", "password".toCharArray());
        MongoClient mongoClient = new MongoClient(new ServerAddress("localhost", 27017), Arrays.asList(credential));
        MongoDatabase db = mongoClient.getDatabase( "bulktest" );

        MongoCollection<Document> col = db.getCollection("test1");

        List<InsertOneModel<Document>> content;

        // iterating on different bulk sizes
        for (int size:TEST_SIZES){
            long total = 0;

            // doing each test a few times to get a more stable average
            for (int a = 0; a < AVG_FACTOR; a++) {

                // creating the 100K documents for insert BEFORE the benchmark, so that we won't deal with document creation as a part of the test
                content = new ArrayList<>();
                for (int d = 0; d < DOC_COUNT; d++) {
                    content.add(new InsertOneModel<>(createSampleDocument()));
                }
                Iterator<InsertOneModel<Document>> iterator = content.iterator();

                // each 'testBulkWrite' returns total insert time of 100K documents in the bulk size as passed in the parameter. We sum these times.
                total += testBulkWrite(col, size, iterator);
            }

            System.out.println(String.format("%d", total / AVG_FACTOR));

        }
    }

    private static long testBulkWrite(MongoCollection<Document> col, int bulkSize, Iterator<InsertOneModel<Document>> iterator) {
        long start = System.currentTimeMillis();

        for (int i = 0; i < DOC_COUNT; ){
            ArrayList<InsertOneModel<Document>> bulk = new ArrayList<>();

            for(int j = 0; j < bulkSize && i < DOC_COUNT; j++,i++){
                bulk.add(iterator.next());
            }

            col.bulkWrite(bulk, new BulkWriteOptions().ordered(false));
        }
        return (System.currentTimeMillis() - start);
    }

    private static Document createSampleDocument(){
        Document doc = new Document();
        doc.append("fname", "My");
        doc.append("lname", "Name");
        for (int i = 0; i < 3; i++){
            doc.append("text" + i, "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.");
        }
        return doc;
    }

还有其他固有的瓶颈吗？所以我想念什么？

谢谢

MongoDB 3.6-4.0批量插入限制

0 个答案: