Question

我正在使用Node的stream.pipeline功能将一些数据上传到S3。我实现的基本思想是从请求中提取文件并将其写入S3。我有一个pipeline，可以拉出zip文件并将其成功写入S3。但是，我希望我的第二个pipeline发出相同的请求，但是解压缩并将解压缩的文件写入S3。管道代码如下所示：

pipeline(request.get(...), s3Stream(zipFileWritePath)),
pipeline(request.get(...), new unzipper.Parse(), etl.map(entry => entry.pipe(s3Stream(createWritePath(writePath, entry)))))

s3Stream函数如下所示：

function s3Stream(file) {
    const pass = new stream.PassThrough()
    s3Store.upload(file, pass)
    return pass
}

第一个pipeline运行良好，目前在生产中运行良好。但是，添加第二条管道时，出现以下错误：

Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
at Parse.onclose (internal/streams/end-of-stream.js:56:36)
at Parse.emit (events.js:187:15)
at Parse.EventEmitter.emit (domain.js:442:20)
at Parse.<anonymous> (/node_modules/unzipper/lib/parse.js:28:10)
at Parse.emit (events.js:187:15)
at Parse.EventEmitter.emit (domain.js:442:20)
at finishMaybe (_stream_writable.js:641:14)
at afterWrite (_stream_writable.js:481:3)
at onwrite (_stream_writable.js:471:7)
at /node_modules/unzipper/lib/PullStream.js:70:11
at afterWrite (_stream_writable.js:480:3)
at process._tickCallback (internal/process/next_tick.js:63:19)

任何想法可能导致此问题或解决方案解决此问题，将不胜感激！

Answer 1

TL; DR

当您使用管道接受完全消耗可读流时，您不希望在可读流结束之前停止任何操作。

深潜

与这些恶作剧者一起工作一段时间后，这里提供了一些更有用的信息。

import stream from 'stream'

const s1 = new stream.PassThrough()
const s2 = new stream.PassThrough()
const s3 = new stream.PassThrough()

s1.on('end', () => console.log('end 1'))
s2.on('end', () => console.log('end 2'))
s3.on('end', () => console.log('end 3'))
s1.on('close', () => console.log('close 1'))
s2.on('close', () => console.log('close 2'))
s3.on('close', () => console.log('close 3'))

stream.pipeline(
    s1,
    s2,
    s3,
    async s => { for await (_ of s) { } },
    err => console.log('end', err)
)

现在如果我打电话给s2.end()，它将关闭所有父母

end 2
close 2
end 3
close 3

管道等于s3（s2（s1）））

但是如果我调用s2.destroy()，它会打印并销毁所有内容，这是您的问题，在此流在正常结束之前已被销毁，无论是错误还是asyncGenerator / asyncFunction中的return / break / throws

close 2
end Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
    at PassThrough.onclose (internal/streams/end-of-stream.js:117:38)
    at PassThrough.emit (events.js:327:22)
    at emitCloseNT (internal/streams/destroy.js:81:10)
    at processTicksAndRejections (internal/process/task_queues.js:83:21) {
  code: 'ERR_STREAM_PREMATURE_CLOSE'
}
close 1
close 3

您不能让任何一个流都无法捕获其错误

stream.pipeline（）在调用allback后将悬空的事件侦听器留在流上。如果发生故障后重用流，则可能导致事件侦听器泄漏和误吞的错误。

node source (14.4)

  const onclose = () => {
    if (readable && !readableEnded) {
      if (!isReadableEnded(stream))
        return callback.call(stream, new ERR_STREAM_PREMATURE_CLOSE());
    }
    if (writable && !writableFinished) {
      if (!isWritableFinished(stream))
        return callback.call(stream, new ERR_STREAM_PREMATURE_CLOSE());
    }
    callback.call(stream);
  };

错误[ERR_STREAM_PREMATURE_CLOSE]：节点管道流中过早关闭

1 个答案:

TL; DR

深潜