Question

我正在尝试使用包含大约145,000个JSON对象的数组运行猫鼬insert many命令。我在Mac测试环境中本地运行了此命令，但是当我在开发Windows环境中运行它时，它并没有崩溃。

我尝试运行“ node max-old-space = 6086 app.js”命令，该命令在我的mac上运行良好（无崩溃），但在具有相同对象数组的Windows 2016机器上崩溃。这是Windows问题吗？

我还用“ something”一词替换了下面的敏感值，以不透露我的应用程序在做什么。

const folder = "./monthly/"
const output = []
var count = 0;
const csvFilePath = folder + req.body.fileName

csv()
.fromFile(csvFilePath)
.on('error', (err) => {
  console.log(err);
  res.send({success: false, message: err.message})
})
.then((jsonObj) => {

  jsonObj.forEach(function(value) {

    var originalDate = new Date(value['something'])
    var something= new Date(originalDate.setMonth(originalDate.getMonth() + 2));

    const body = new Report ({
      something: value['something'],
      something: value['something'],
      something: value['something'],
      something: value['something'],
      something: value['something'],
      something: value['something'],
      something: value['something'],
      something: value['something'],
      something: value['something'],
      something: value['something'],
      something: value['something'],
      something: value['something'],
      something: value['something'],
      something: value['something'],
      something: value['something'],
      something: value['something'],
      something: value['something'],
      something: value['something'],
      something: value['something'],
      something: value['something'],
      something: value['something'],
      something: something.toLocaleDateString(),
      something: req.decoded.something
    })

    output.push(body);


  })

  //Everything up until this point runs perfectly 
  //This is where it crashes 
  Report.insertMany(output, function (err, results) {
    if (err) {
      console.log(err)
      res.send({success: false, message: err.message})
    } else {
      console.log(results);
      res.send({success: true, message: "Report Submitted"})
    }
  })
})

Answer 1

如果要使用猫鼬和节点执行此操作，则需要开始将对象批处理为较小的插入。使用async/await可能类似于：

let toInsert = [];
for (let i = 0; i < items.length; i++) {
  toInsert.push(new Report(item...));
  const isLastItem = i === items.length - 1;
  // every 100 items, insert into the database
  if (i % 100 === 0 || isLastItem) {
    await Report.insertMany(toInsert);
    toInsert = [];
  }
}

猫鼬插入很多堆内存

1 个答案: