Question

我正在尝试编写程序从s3获取zip文件，解压缩，然后将其上传到S3。但我发现了两个我无法抓住的例外情况。

1。 StreamContentLengthMismatch: Stream content length mismatch. Received 980323883 of 5770104761 bytes.这种情况不规律地发生。

2. NoSuchKey: The specified key does not exist.当我输入错误的密钥时会发生这种情况。

当出现这两个异常时，此程序崩溃。

我想正确地抓住并处理这两个例外。

我想防止崩溃。

   const unzipUpload = () => {
        return new Promise((resolve, reject) => {
            let rStream = s3.getObject({Bucket: 'bucket', Key: 'hoge/hoge.zip'})
                .createReadStream()
                    .pipe(unzip.Parse())
                    .on('entry', function (entry) {
                        if(entry.path.match(/__MACOSX/) == null){

                            // pause
                            if(currentFileCount - uploadedFileCount > 10) rStream.pause()

                            currentFileCount += 1
                            var fileName = entry.path;
                            let up = entry.pipe(uploadFromStream(s3,fileName))

                            up.on('uploaded', e => {
                                uploadedFileCount += 1
                                console.log(currentFileCount, uploadedFileCount)

                                //resume
                                if(currentFileCount - uploadedFileCount <= 10) rStream.resume()

                                if(uploadedFileCount === allFileCount) resolve()
                                entry.autodrain()
                            }).on('error', e => {
                                reject()
                            })
                        }

                    }).on('error', e => {
                        console.log("unzip error")
                        reject()
                    }).on('finish', e => {
                        allFileCount = currentFileCount
                    })
            rStream.on('error', e=> {
                console.log(e)
                reject(e)
            })
        })
    }

    function uploadFromStream(s3,fileName) {
        var pass = new stream.PassThrough();

        var params = {Bucket: "bucket", Key: "hoge/unzip/" + fileName, Body: pass};
        let request = s3.upload(params, function(err, data) {
            if(err) pass.emit('error')
            if(!err) pass.emit('uploaded')
        })
        request.on('httpUploadProgress', progress => {
            console.log(progress)
        })

        return pass
    }

这是解压缩时使用的库。 https://github.com/mhr3/unzip-stream

帮助我!!

Answer 1

如果您想抓住NoSuchKey引发的createReadStream错误，您有两个选择：

在阅读之前检查密钥是否存在。
从流中捕捉错误

<强>第一：

s3.getObjectMetadata(key)
  .promise()
  .then(() => {
    // This will not throw error anymore
    s3.getObject().createReadStream();
  })
  .catch(error => {
    if (error.statusCode === 404) {
      // Catching NoSuchKey
    }
  });

唯一一种情况，如果在解析来自getObjectMetadata的响应和运行createReadStream

第二次：

s3.getObject().createReadStream().on('error', error => {
    // Catching NoSuchKey & StreamContentLengthMismatch
});

这是一种更通用的方法，可以捕获所有其他错误，例如网络问题。

Answer 2

您需要提前侦听发出的错误。您的错误处理程序仅在解压缩部分期间查找错误。

脚本的简化版本。

s3.getObject(params)
.createReadStream()
.on('error', (e) => {
  // handle aws s3 error from createReadStream
})
.pipe(unzip)
.on('data', (data) => {
  // retrieve data
})
.on('end', () => {
  // stream has ended
})
.on('error', (e) => {
  // handle error from unzip
});

这样，您无需再向AWS发出通知，以确定它是否存在。

Answer 3

您可以在收到的流中收听事件（如错误，数据，完成）。 Read more on events

function getObjectStream (filePath) { return s3.getObject({ Bucket: bucket, Key: filePath }).createReadStream() } let readStream = getObjectStream('/path/to/file.zip') readStream.on('error', function (error) { // Handle your error here. })

针对“无密钥”错误进行了测试。

it('should not be able to get stream of unavailable object', function (done) { let filePath = 'file_not_available.zip' let readStream = s3.getObjectStream(filePath) readStream.on('error', function (error) { expect(error instanceof Error).to.equal(true) expect(error.message).to.equal('The specified key does not exist.') done() }) })

经过成功测试。

it('should be able to get stream of available object', function (done) { let filePath = 'test.zip' let receivedBytes = 0 let readStream = s3.getObjectStream(filePath) readStream.on('error', function (error) { expect(error).to.equal(undefined) }) readStream.on('data', function (data) { receivedBytes += data.length }) readStream.on('finish', function () { expect(receivedBytes).to.equal(3774) done() }) })

Answer 4

为了防止崩溃，您需要异步侦听对象的头部元数据，它不会返回整个对象，这将花费更少的时间。试试这个！

isObjectErrorExists = async functions () => {
  try {
const s3bucket = {
secret key: '',
client id: ''
}
  const params = {
       Bucket: 'your bucket name',
       Key: 'path to object'
};
    await s3bucket.headObject(params).promise(); // adding promise will let you add await to listen to process untill it completes.

    return true;
  } catch (err) {
      return false; // headObject threw error.
    }
    throw new Error(err.message); 
  }
}

public yourFunction = async() => {
if (await this.isObjectErrorExists()) {
s3Bucket.getObject().createReadStream(); // works smoothly
}
}

s3.getObject（）。createReadStream（）：如何捕获错误？

4 个答案: