Question

我想知道异步处理多个读取流的正确方法是什么，以便能够将结果合并到一个文件中。读取的流来自SFTP服务器上的文件，我正在使用称为ssh2-sftp-client的库进行读取。

这就是我想要做的。

const fs = require('fs');
const Client = require('ssh2-sftp-client');
const concat = require('concat-stream');

const sftp = new Client();
sftp.connect(configObj).then(() => {

  const fileNames = ['file1', 'file2'];
  const streamPromiseArr = [];

  // Loop through the files
  for (fileName of fileNames) {

    // Collect the promises (each promise resolves to a final buffer)
    const streamPromise = new Promise((resolve, reject) => {
      sftp.get(fileName, true, 'utf8').then((stream) => {
        stream
          .pipe(decrypt)
          .pipe(unzipper.Parse())
          .pipe(selectFile)
          .pipe(filterArray)
          .pipe(concat(res => resolve(res)))
      })
    });
    streamPromiseArr.push(streamPromise);
  }

  // Collect all the buffers and write them to a file
  Promise.all(streamPromiseArr).then((buffArr) => {
    const finalBuff = Buffer.concat(buffArr);
    fs.writeFile('consolidate_file.csv', finalBuff, ((err) => {
      if (err) console.log(err);
    }));
  }).then(() => sftp.end());
})

这给了我一个错误：

TypeError: cb is not a function
    at afterWrite (_stream_writable.js:454:3)
    at onwrite (_stream_writable.js:445:7)
    at /Users/jonlamb/projects/node-test/node_modules/unzipper/lib/PullStream.js:59:60
    at afterWrite (_stream_writable.js:454:3)
    at _combinedTickCallback (internal/process/next_tick.js:144:20)
    at process._tickCallback (internal/process/next_tick.js:180:9)

我认为这与我致电sftp.end()有关。我不知道这是为什么，但是如果我将sftp.end()链接到sftp.get()的末尾，我不会收到错误，但是第二次循环的连接已关闭，因此失败。

NodeJS并行运行多个读取流

0 个答案: