扫描S3存储桶,仅将zip文件上传到S3存储桶

时间:2019-06-09 22:52:43

标签: python amazon-web-services amazon-s3 boto3

这是我的代码,用于将文件从onprem服务器上载到S3存储桶,并且当我指定要上传的文件时,该代码有效。

import boto3
import sys, ast
import io
import os
from zipfile import ZipFile
import json
from pprint import pprint
import urllib.request
import traceback
from datetime import datetime


accesskey = sys.argv[1]
secretkey = sys.argv[2]
sessiontoken = sys.argv[3]
kmskeyid = sys.argv[4]
filepath = sys.argv[5]
s3bucket = sys.argv[6]
s3key = sys.argv[7]

try:
    s3_client = boto3.client(
        's3',
        aws_access_key_id = accesskey,
        aws_secret_access_key = secretkey,
        aws_session_token = sessiontoken,
        region_name="us-east-1"
    )
except Exception as e:
    print("[ERROR]\nFAILURE CREATING S3 CLIENT!")
    traceback.print_exc()
    sys.exit(1)

def upload_file_to_s3():
    #s3_client.upload_file(version + '.zip', s3bucket, s3key,ExtraArgs={"ServerSideEncryption": "aws:kms", "SSEKMSKeyId": "arn:aws:kms:us-east-1:XXXX"})
    with open(filepath, 'rb') as code:
        s3_client.upload_fileobj(code, s3bucket, s3key,ExtraArgs={"ServerSideEncryption": "aws:kms", "SSEKMSKeyId": kmskeyid})


try:
    print('Uploading object to S3 bucket: ' + s3bucket + "/" + s3key)
    upload_file_to_s3()
    print("[INFO]\nFile successfully uploaded to S3.")
except Exception as e:
    print("[ERROR]\nFAILURE UPLOADING TO S3!")
    traceback.print_exc()
    sys.exit(1)

我从本地服务器运行的命令:

python3 ./scripts/python/upload_file_to_s3_min.py $AWS_ACCESS_KEY_ID $AWS_SECRET_ACCESS_KEY $AWS_SESSION_TOKEN $AWS_KMS_KEY_ID *.zip* $AWS_BUCKET deploy/scripts/

错误消息:

Uploading object to S3 bucket: file2.zip/file3.zip
[ERROR]
FAILURE UPLOADING TO S3!
Build step 'Execute shell' marked build as failure
Finished: FAILURE

由于某种原因,脚本未传递$ bucketname

谢谢

1 个答案:

答案 0 :(得分:0)

一些一般性反馈...

人们通常将其AWS凭证存储在配置文件中。这可以通过使用AWS Command-Line Interface (CLI)并运行aws configure命令来完成。

以这种方式保存凭据后,您的代码就无需引用任何凭据。它们将被自动发现。

因此,您的代码可能很简单:

s3_client = boto3.client('s3', region_name="us-east-1")

with open(filepath, 'rb') as code:
    s3_client.upload_fileobj(code, s3bucket, s3key,ExtraArgs={"ServerSideEncryption": "aws:kms", "SSEKMSKeyId": kmskeyid})

如果您只是想从命令行中调用它,那么不用编写自己的脚本,只需使用AWS CLI命令即可:

aws s3 cp foo.zip s3://my-bucket/foo.zip --sse-kms-key-id xxx