Question

我正在尝试在Node Js中创建一个CSV文件（使用fast-csv），但即使在使用--max-old-space-size = 2000000之后，我也会收到致命错误：CALL_AND_RETRY_LAST分配失败，因为有没有大小（比如每个大小为1.5mb的3000）。

节点js按逻辑顺序创建每个文件，但为什么在创建时大小为零，并且在生成所有文件后分配内存

单独创建文件和分配内存的任何可能方法？

提前致谢

Answer 1

尝试增加内存限制：

node --max-old-space-size=8192 fileName.js

或者使用像Mongodb这样的数据库。

Answer 2

实际上，创建CSV并不难。在创建大型CSV文件时，最好将每一行写到一个文件中，而不是先在内存中创建所有内容。

这是我创建（大）CSV文件的代码：

const fs = require('fs');
const path = require('path');

const separator = ',';

var data = [
    {name: 'Sam'  , age: 33.5, place: 'some place, with a separator in'},
    {name: 'Sofie', age: 31  , place: 'some place with "double quotes" in'},
    {name: 'Nic'  , age: 1   , place: 'some place with\na newline in'}
];

// construct csv output stream:
const outputPath = path.join(__dirname, '/test.csv');
const output = fs.createWriteStream(outputPath, { encoding: 'utf8' });

// add separator indication (So Excel knows what the CSV separator is):
output.write(`sep=${separator}\n`);

// if no data, end creation of file:
if(data.length == 0) {
    output.end(() => {
        console.log('done');
    });
    return;
}

// get headers from first entry:
var headers = Object.keys(data[0]);

// write headers to file:
output.write(`${constructCsvLine(headers)}\n`);

// write the rest of the data:
for (var i = 0; i < data.length; i++) {
    var entry = data[i];

    var line = [];

    // only get fields that we have in our headers:
    for (var j = 0; j < headers.length; j++) {
        var key = headers[j];

        var field = entry[key];
        if(field === undefined) field = '';

        line.push(field);
    }

    output.write(`${constructCsvLine(line)}\n`);
}

output.end(() => {
    console.log('done');
});



function constructCsvLine(fields) {
    var encodedFields = [];
    for (var i = 0; i < fields.length; i++) {
        var field = '' + fields[i];

        if(field.includes('"')) field = `"${field.replace(/\"/g, '""')}"`; // replace single quotes with double quotes and enclose in quotes
        if(field.includes(separator)) field = `"${field}"`; // enclose in quotes when separator occurs in field
        if(field.includes("\n")) field = `"${field}"`; // enclose in quotes when newline occrus in field
        field = field.replace(/\n/g, "\r"); // seems to be parsed correctly as newline by Excel

        encodedFields.push(field);
    }
    return encodedFields.join(separator);
}

节点JS创建大量没有CSV文件而没有内存不足错误

2 个答案: