Question

Mongo Version: 3.6 or greater
Java Version : 1.8

I have close to a 10 Million records that I receive from a change stream daily if the document is new I need to insert the document and in case the document already exists I need to push the updates to the array and increment the version number field all this has to be done as a bulk operation I have almost figured out every things except the version number inside the array.


I have been able to achieve the same when I do a round trip to mongo but I need to achieve this in a Bulk operation for a batch size of 10K I just want one mongo db hit .




How do I get the value of the outer version into the inner one every time I push something into the array.

 The details of the code are below.

Bson filterQuery = new Document("_id", "1");

 Bson updateQuery = new Document("$push", getFormattedDocumentUpdate(db));

((Document) updateQuery).append("$setOnInsert", getFormattedDocumentInsert("1", 1));

writes.add(new UpdateOneModel<Document>(filterQuery, updateQuery, (new UpdateOptions()).upsert(true)));

Bson filterQueryAppended = new Document();

((Document) filterQueryAppended).put("_id", "1");

 BasicDBObject incValue = new BasicDBObject("Outer_Version", 1); 

 BasicDBObject intModifier = new BasicDBObject("$inc", incValue);

 UpdateOneModel model = new UpdateOneModel<Document>(filterQueryAppended, intModifier);

  writes.add(model);

//how do I refer the outer count when I push into an array ?


This is how the structure looks like (Pretty much a history of documents)

id : 12345
Outer_Version : 1
{
    //this is the first snapshot inserted this doesn't change
    A : 1
    B : 2
    C:  3
}
delta[ 
       { 
          A : 1,
          B : 2, 
          C: 3 , 
          Inner_Version : 1
        }
     ]   //means 1


Update 1:

id:12345
Outer_Version : 2
{
    //this is the first snapshot inserted this doesn't change
    A : 1
    B : 2
    C : 3
}
delta [ 
        { 
           A : 1, 
           B : 2, 
           C: 3 , 
           Inner_Version : 1 
        } , //this 2
        {
           B : 45,
          Inner_Version : 2
         }
      ]

我无法获取数组的大小，因为这是一个移动的列表每隔几天即可访问Amazon S3。

我不能假设它是一个数组，因为它会有一个索引，所以我可以加上一个大小。

我无法访问mongo集合，因为那样会降低性能。

我需要某种方式可以引用正在使用的集合中的外部计数器

在mongo数组中增加外部版本号

0 个答案: