Question

周末快乐，

我正在执行一个任务，该任务从API提取数据，然后将其存储到csv文件中，然后从那里直接上传到AWS S3存储桶。我尝试了几种方法，但目前停留在最后一点。任何帮助将不胜感激

下面的代码将演示大多数问题以及到目前为止我一直在尝试的事情。

首先，我将从API获取数据

async systems() {
        const endpoint = sampleEndPoints.SYSTEMS
        return this.aggregateEndpoint(endpoint)
    }

第二，我将获取回取的数据并将其放入csv文件中作为缓冲区。（因为稍后我必须将它们存储在fs.createReadStream中）

// generate JSON to Buffer
    async generateCsvToBuffer(json){
            const {aws} = this.config
            var ws = xlsx.utils.json_to_sheet(json)

            var wb = xlsx.utils.book_new();

          await xlsx.utils.book_append_sheet(wb, ws, 'Systems')

            const csvParsed = xlsx.write(wb, { type: 'buffer'})


            return csvParsed;
    }

第三，我从那个buffer data得到一个csvParsed以便upload到亚马逊AWS S3。问题就在这里，Body: fileStream.path应该显示文件的内容，但不幸的是，它像这样从fs.createReadStream

'{"type":"Buffer","data":[80,75,3,4,10,0,0,0,0,0,249,117,199,78,214,146,124

 async uploadSample(file){
        const {aws} = this.config

        AWS.config.update({
          secretAccessKey: aws.secretAccessKey,
          accessKeyId: aws.accessKeyId,
          region: 'us-east-2'
        })


        const bufferObject = new Buffer.from(JSON.stringify(file))


     /*** WE NEED THE FILE SYSTEM IN ORDER TO STORE  */
        const fileStream = fs.createReadStream(bufferObject)


        const uploadParams = {Bucket: aws.bucket, Key: aws.key, Body: fileStream.path}


        const s3 = new AWS.S3()
       await s3.upload(uploadParams,null,function(error, file){
          if(error){
            console.log(error)
          } else {
            console.log('Successfully uploaded')
          }
        })
    }

我所有的功能都将在server.js中执行。因此，如果您对此有个了解，那么您实际上可以了解问题的全部情况

app.get('/systems/parsed', async(req, res) => {
    const Sample = await Sample()

//Fetch the data from an API 
    const systems = await Cache.remember('systems', async() => {
        return Sample.systems()
    })
    const integration = await IntegrationInstance()

/** GET THE RESPONSE DATA AND PUT THEM IN A CSV FILE*/
    const result = await integration.generateCsvToBuffer(systems)

    const aws = await AwsInstance()

/*** GET THE SYSTEMS FILE (CSV FILE) THEN UPLOAD THEM INTO THE AWS S3 BUCKET*/

    const awsUpload = await aws.uploadWorkedWithBuffer(result)


    return res.send(awsUpload);
})

我在这里唯一关心的是，文件已成功上传到Amazon AWS S3，但文件的内容仍在Buffer中。对现有功能的任何帮助/任何较短的方式都将不胜感激。
这是我的总结：从服务器获取数据->从Web浏览器将Csv文件作为缓冲区BUT放入->然后从那里上传到Amazon AWS S3存储桶->问题是文件已上传但文件的内容仍在缓冲区中。

Answer 1

看来您正在使事情变得比这里必要的复杂。根据{{3}}文档，您可以直接将缓冲区传递给上载，而不用从缓冲区创建流。我怀疑您的根本问题是通过流而不是流本身传递路径。

Answer 2

我实际上解决了它。

首先，每当创建函数generateCsvToBuffer时，请记住在bookType（工作簿）上有一个wb，以便s3识别它。该功能应该是这样的

    async generateCsvToBuffer(json){
            const {aws} = this.config
            var ws = xlsx.utils.json_to_sheet(json)

            var wb = xlsx.utils.book_new();

          await xlsx.utils.book_append_sheet(wb, ws, 'Systems')

            const csvParsed = xlsx.write(wb, { type: 'buffer', bookType: 'csv'})

            return csvParsed;
    }

第二，您必须将Content-Disposition: attachment导入到uploadParams中以进行Aws配置

 async uploadSample(file){
        const {aws} = this.config

        AWS.config.update({
          secretAccessKey: aws.secretAccessKey,
          accessKeyId: aws.accessKeyId,
          region: 'us-east-2'
        })


        const bufferObject = new Buffer.from(JSON.stringify(file))


     /*** WE NEED THE FILE SYSTEM IN ORDER TO STORE  */
        const fileStream = fs.createReadStream(bufferObject)


        const uploadParams = {Bucket: aws.bucket, Key: aws.key, Body: fileStream.path}


        const s3 = new AWS.S3()
       await s3.upload(uploadParams,null,function(error, file){
          if(error){
            console.log(error)
          } else {
            console.log('Successfully uploaded')
          }
        })
    }

直接从服务器将CSV文件上传到AWS S3存储桶

2 个答案: