从EC2实例

时间:2016-11-10 21:01:42

标签: curl amazon-s3 amazon-ec2

我有一个从AWS S3下载文件的脚本,它可以在AWS外部的单独服务器上运行。但是,当我将此脚本放在EC2实例上并尝试它时,它会返回错误“SignatureDoesNotMatch - 我们计算的请求签名与您提供的签名不匹配。检查您的密钥和签名方法。”

但它适用于其他服务器。 ec2与s3主机位于同一区域。我猜这与主持人有关。我尝试了这些host / url,但它返回相同的错误。

任何人都可以使用curl从ec2下载s3文件吗?我只需要使用curl。如果您知道如何通过卷曲来回答,请回答。感谢。

https://s3-ap-southeast-1.amazonaws.com/ $桶​​/ $文件

https://s3.amazonaws.com/ $桶​​/ $文件

的https://$bucket.s3-ap-southeast-1.amazonaws.com/$file

的https://$bucket.s3.amazonaws.com/$file

#!/bin/sh
file="file-name"
bucket="bucket-name"
resource="/${bucket}/${file}"
contentType="application/x-compressed-tar"
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`"
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key="xxxxxxxxxxxxxxxxxx"
s3Secret="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
signature=$(echo -en "${stringToSign}" | openssl sha1 -hmac "${s3Secret}" -binary | base64)
curl -H "Host: s3-ap-southeast-1.amazonaws.com" \
 -H "Date: $dateValue" \
 -H "Content-Type: $contentType" \
 -H "Authorization: AWS ${s3Key}:${signature}" \
 https://s3-ap-southeast-1.amazonaws.com/$bucket/$file -o $file

3 个答案:

答案 0 :(得分:2)

不要编写CURL命令,而应考虑使用AWS Command-Line Interface (CLI)

它有一个aws s3 cp命令,可以将内容复制到Amazon S3存储桶中(甚至在存储桶之间)。

答案 1 :(得分:1)

Titi Wangsa bin Damhore的回答非常有帮助。这是一个稍微修改的版本,可以使用分配给EC2实例的IAM配置文件中的凭据。我也没有在计算机上使用xxd,因此对其进行了调整以使用sed。

#!/bin/bash

#### Variables

REGION=eu-west-2
S3_KEY='/some/file.txt'   # The file you want to download
FILENAME=$(basename $S3_KEY)
S3_HOST=yourbucket.s3.amazonaws.com # Endpoint for the S3 bucket


#### Get Temporary AWS Credentials from instance metadata ###### 

ROLE_NAME=$(curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/)
CREDENTIALS_TEXT=$(curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/$ROLE_NAME)

S3_SECRET_KEY=$(echo $CREDENTIALS_TEXT | sed -e 's/.*SecretAccessKey" : "\(.*\)", "Token.*/\1/')
S3_ACCESS_KEY=$(echo $CREDENTIALS_TEXT | sed -e 's/.*AccessKeyId" : "\(.*\)", "SecretAccessKey.*/\1/')
TOKEN=$(echo $CREDENTIALS_TEXT | sed -e 's/.*Token" : "\(.*\)", "Expiration.*/\1/')

####################################

##### Create the string that needs signing ########

AMZ_DATE=$(date -u "+%Y%m%dT%H%M%SZ")
DATE=$(/bin/echo ${AMZ_DATE} | cut -b 1-8)

CANONICAL_REQUEST="GET\n${S3_KEY}\n\nhost:${S3_HOST}\nx-amz-content-sha256:UNSIGNED-PAYLOAD\nx-amz-date:${AMZ_DATE}\nx-amz-security-token:${TOKEN}\n\nhost;x-amz-content-sha256;x-amz-date;x-amz-security-token\nUNSIGNED-PAYLOAD"
STRING_TO_SIGN="AWS4-HMAC-SHA256\n${AMZ_DATE}\n${DATE}/${REGION}/s3/aws4_request\n$(/bin/echo -en ${CANONICAL_REQUEST} | sha256sum | cut -f 1 -d' ')"

####################################

#### Use Temporary AWS Credentials to create signing key ###### 

AWS_SIG_V4_AND_S3_SECRET_KEY=AWS4${S3_SECRET_KEY}

DATE_HMAC_HEX=$(/bin/echo -n "${DATE}" | openssl sha256 -hmac "${AWS_SIG_V4_AND_S3_SECRET_KEY}" | cut -f 2 -d' ')
DATE_HMAC_BIN=$(/bin/echo -n "${DATE_HMAC_HEX}" | sed 's/\([0-9A-F]\{2\}\)/\\\\\\x\1/gI' | xargs printf)

REGION_HMAC_HEX=$(/bin/echo -n "${REGION}"  | openssl sha256 -hmac "${DATE_HMAC_BIN}" | cut -f 2 -d' ')
REGION_HMAC_BIN=$(/bin/echo -n "${REGION_HMAC_HEX}" | sed 's/\([0-9A-F]\{2\}\)/\\\\\\x\1/gI' | xargs printf)

SERVICE_HMAC_HEX=$(/bin/echo -n "s3"  | openssl sha256 -hmac "${REGION_HMAC_BIN}" | cut -f 2 -d' ')
SERVICE_HMAC_BIN=$(/bin/echo -n "${SERVICE_HMAC_HEX}" | sed 's/\([0-9A-F]\{2\}\)/\\\\\\x\1/gI' | xargs printf)

SIGNING_KEY_HEX=$(/bin/echo -n "aws4_request"  | openssl sha256 -hmac "${SERVICE_HMAC_BIN}" | cut -f 2 -d' ')
SIGNING_KEY_BIN=$(/bin/echo -n "${SIGNING_KEY_HEX}" | sed 's/\([0-9A-F]\{2\}\)/\\\\\\x\1/gI' | xargs printf)

####################################

#### Sign string with signing key ## 

SIGNATURE_HEX=$(/bin/echo -ne "${STRING_TO_SIGN}" | openssl sha256 -hmac "${SIGNING_KEY_BIN}" | cut -f 2 -d' ')

####################################

#### Put it all together into a curl request ## 

curl https://${S3_HOST}${S3_KEY} \
    -H "Host: ${S3_HOST}" \
    -H "x-amz-content-sha256: UNSIGNED-PAYLOAD" \
    -H "x-amz-date: ${AMZ_DATE}" \
    -H "X-Amz-Security-Token: ${TOKEN}" \
    -H "Authorization: AWS4-HMAC-SHA256 Credential=${S3_ACCESS_KEY}/${DATE}/${REGION}/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date;X-Amz-Security-Token,Signature=${SIGNATURE_HEX}" -o $FILENAME

####################################

答案 2 :(得分:0)

试试这个......

#!/bin/sh
S3_KEY='/file.txt'
S3_SECRET_KEY='xxx'
S3_ACCESS_KEY='AKIAxxx'
S3_HOST=yourbucket.s3.amazonaws.com

AMZ_DATE=$(date -u "+%Y%m%dT%H%M%SZ")
DATE=$(/bin/echo ${AMZ_DATE} | cut -b 1-8)
REGION=ap-southeast-1
CANONICAL_REQUEST="GET\n${S3_KEY}\n\nhost:${S3_HOST}\nx-amz-content-sha256:UNSIGNED-PAYLOAD\nx-amz-date:${AMZ_DATE}\n\nhost;x-amz-content-sha256;x-amz-date\nUNSIGNED-PAYLOAD"
STRING_TO_SIGN="AWS4-HMAC-SHA256\n${AMZ_DATE}\n${DATE}/${REGION}/s3/aws4_request\n$(/bin/echo -en ${CANONICAL_REQUEST} | sha256sum | cut -f 1 -d' ')"

AWS_SIG_V4_AND_S3_SECRET_KEY=AWS4${S3_SECRET_KEY}

DATE_HMAC_HEX=$(/bin/echo -n "${DATE}" | openssl sha256 -hmac "${AWS_SIG_V4_AND_S3_SECRET_KEY}" | cut -f 2 -d' ')
DATE_HMAC_BIN=$(/bin/echo -n "${DATE_HMAC_HEX}" | xxd -r -p)

REGION_HMAC_HEX=$(/bin/echo -n "${REGION}"  | openssl sha256 -hmac "${DATE_HMAC_BIN}" | cut -f 2 -d' ')
REGION_HMAC_BIN=$(/bin/echo -n "${REGION_HMAC_HEX}" | xxd -r -p)

SERVICE_HMAC_HEX=$(/bin/echo -n "s3"  | openssl sha256 -hmac "${REGION_HMAC_BIN}" | cut -f 2 -d' ')
SERVICE_HMAC_BIN=$(/bin/echo -n "${SERVICE_HMAC_HEX}" | xxd -r -p)

SIGNING_KEY_HEX=$(/bin/echo -n "aws4_request"  | openssl sha256 -hmac "${SERVICE_HMAC_BIN}" | cut -f 2 -d' ')
SIGNING_KEY_BIN=$(/bin/echo -n "${SIGNING_KEY_HEX}" | xxd -r -p)


SIGNATURE_HEX=$(/bin/echo -ne "${STRING_TO_SIGN}" | openssl sha256 -hmac "${SIGNING_KEY_BIN}" | cut -f 2 -d' ')

curl -vv https://${S3_HOST}/file.txt \
    -H "Host: ${S3_HOST}" \
    -H "x-amz-content-sha256: UNSIGNED-PAYLOAD" \
    -H "x-amz-date: ${AMZ_DATE}" \
    -H "Authorization: AWS4-HMAC-SHA256 Credential=${S3_ACCESS_KEY}/${DATE}/${REGION}/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=${SIGNATURE_HEX}"

这是使用AWS Signature V4。 我测试了这个,它适用于CentOS 7.3和Ubuntu LTS 16.04,适用于我在东南1区的水桶。 我强烈建议使用John Rotenstein的建议。