Question

我一直在苦苦挣扎，寻找了很长时间。我知道有关于此的答案，但都没有效果。

我为此使用了fs.createReadStream和readLine。但它正在使用fs.close（）关闭文件读取。因此在缓冲区上使用它时根本不起作用。所有文件的读取都无法中断它......

然后我用了这个：

const stream = require('stream');

let bufferStream = new stream.PassThrough();
bufferStream.end(hexaviaFile.buffer);

bufferStream
    .pipe(require('split')())
    .pipe(es.mapSync(function(line){

        // pause the readstream
        bufferStream.pause();

        // DO WHATEVER WITH YOUR LINE
        console.log('line content = ' + line);

        // resume the readstream, possibly from a callback
        bufferStream.resume();

    }).on('error', function(err){
        console.log('Error while reading file.' + err);
    }).on('end', function(){
        console.log('end event !');

    }).on('close', function(){
        console.log('close event !');

    })
);

// toString() Failed

我得到[toString（）失败]错误并搜索了它，显然当缓冲区大于节点缓冲区最大大小时出现。

所以我查了一下：

var buffer = require('buffer');
console.log('buffer.kMaxLength = ',  buffer.kMaxLength); // 2147483647
console.log('hexaviaFile.buffer.byteLength = ',  hexaviaFile.buffer.byteLength); // => 413567671

情况并非如此，因为您可以看到提供的数字：
* maxBuffer size = 2Go
*我的缓冲区= 0.4Go

我也试过一些不同的库来做到这一点但是：
1.我希望尽可能降低内存使用率我需要这个阅读才能完全同步。换句话说，我在文件读取后有一些处理，我需要在进行下一步之前完成所有阅读。

我不知道该怎么做:)任何一种（帮助）赞赏

问候。

Answer 1

我忘记了这篇文章。我找到了一种无误的方法来实现这一目标。

这里给出了：https://github.com/request/request/issues/2826

1st创建一个拆分器来读取字符串块

class Splitter extends Transform {
    constructor(options){
        super(options);
        this.splitSize = options.splitSize;
        this.buffer = Buffer.alloc(0);
        this.continueThis = true;
    }
    stopIt() {
        this.continueThis = false;
    }

    _transform(chunk, encoding, cb){

        this.buffer = Buffer.concat([this.buffer, chunk]);

        while ((this.buffer.length > this.splitSize || this.buffer.length === 1) && this.continueThis){
            try {
                let chunk = this.buffer.slice(0, this.splitSize);

                this.push(chunk);
                this.buffer = this.buffer.slice(this.splitSize);
                if (this.buffer[0] === 26){
                    console.log('EOF : ' + this.buffer[0]);
                }
            } catch (err) {
                console.log('ERR OCCURED => ', err);
                break;
            }
        }
        console.log('WHILE FINISHED');
        cb();
    }
}

然后将其传输到您的信息流：

let bufferStream = new stream.PassThrough();
bufferStream.end(hugeBuffer);
let splitter = new Splitter({splitSize : 170}); // In my case I have 170 length lines, so I want to process them line by line
let lineNr = 0;
bufferStream
      .pipe(splitter)
      .on('data', async function(line){

          line = line.toString().trim();

          splitter.pause(); // pause stream so you can perform long time processing with await
          lineNr++;

         if (lineNr === 1){
              // DO stuff with 1st line

         } else {
              splitter.stopIt(); // Break the stream and stop reading so we just read 1st line
         }

         splitter.resume() // resumestream so you can process next chunk
    }).on('error', function(err){
            console.log('Error while reading file.' + err);
            // whatever
     }).on('end', async function(){
           console.log('end event');

           // Stream has ended, do whatever...

    });

NodeJS - 逐行读取缓冲区=＆gt; toString（）失败了

1 个答案: