我有一个最小的python脚本,尝试使用boto将s3对象读入文件。
虽然我能够在传输(GUI)中使用相同的access_key / secret / bucketname / key来获取数据,但我收到了授权错误。这让我相信我在一个非常简单的脚本中做错了什么,但我无法弄清楚是什么。
我在boto中打开了详细调试。它报告的“路径”不包括存储桶名称。它会报告一个auth_path,它^包含存储桶名称。我不知道这是否是一个线索,但这对我来说还不够。
这是极简主义剧本:
boto.set_stream_logger('boto') # turn on lots of debugging
bucket_name = 'chegg-prod-datalake'
key_name = 'taxonomy_subject/mappings/bisac-tutors/subject_maps.json'
print("----- getting connection")
conn = S3Connection(aws_access_key_id, aws_secret_access_key)
print("----- getting bucket")
bucket = conn.get_bucket(bucket_name)
print("----- getting key")
key = bucket.get_key(key_name)
print("----- getting contents")
key.get_contents_to_filename('subject_maps.json')
这是输出:
----- getting connection
2016-06-04 16:45:28,630 boto [DEBUG]:Using access key provided by client.
2016-06-04 16:45:28,630 boto [DEBUG]:Using secret key provided by client.
----- getting bucket
2016-06-04 16:45:28,630 boto [DEBUG]:path=/
2016-06-04 16:45:28,630 boto [DEBUG]:auth_path=/chegg-prod-datalake/
2016-06-04 16:45:28,630 boto [DEBUG]:Method: HEAD
2016-06-04 16:45:28,631 boto [DEBUG]:Path: /
2016-06-04 16:45:28,631 boto [DEBUG]:Data:
2016-06-04 16:45:28,631 boto [DEBUG]:Headers: {}
2016-06-04 16:45:28,631 boto [DEBUG]:Host: chegg-prod-datalake.s3.amazonaws.com
2016-06-04 16:45:28,631 boto [DEBUG]:Port: 443
2016-06-04 16:45:28,631 boto [DEBUG]:Params: {}
2016-06-04 16:45:28,631 boto [DEBUG]:establishing HTTPS connection: host=chegg-prod-datalake.s3.amazonaws.com, kwargs={'port': 443, 'timeout': 70}
2016-06-04 16:45:28,633 boto [DEBUG]:Token: None
2016-06-04 16:45:28,633 boto [DEBUG]:StringToSign:
HEAD
Sat, 04 Jun 2016 23:45:28 GMT
/chegg-prod-datalake/
2016-06-04 16:45:28,633 boto [DEBUG]:Signature:
AWS AKIAJJX6GYAXTBPLRPTA:W0M4MqIYdNwUGI0t4csuCbty3B0=
2016-06-04 16:45:28,633 boto [DEBUG]:Final headers: {'Date': 'Sat, 04 Jun 2016 23:45:28 GMT', 'Content-Length': '0', 'Authorization': u'AWS AKIAJJX6GYAXTBPLRPTA:W0M4MqIYdNwUGI0t4csuCbty3B0=', 'User-Agent': 'Boto/2.37.0 Python/2.7.10 Darwin/14.5.0'}
2016-06-04 16:45:29,342 boto [DEBUG]:Response headers: [('x-amz-bucket-region', 'us-west-2'), ('x-amz-id-2', 'yQdJo0QUHq8KZdbnx30rkSWlhOirrx4d87fHKjMzUf1bfvOu8dY3zxuHh8Qcf5S2mBCz+e1X8Q0='), ('server', 'AmazonS3'), ('transfer-encoding', 'chunked'), ('x-amz-request-id', '4859B765113832F9'), ('date', 'Sat, 04 Jun 2016 23:45:30 GMT'), ('content-type', 'application/xml')]
----- getting key
2016-06-04 16:45:29,342 boto [DEBUG]:path=/taxonomy_subject/mappings/bisac-tutors/subject_maps.json
2016-06-04 16:45:29,342 boto [DEBUG]:auth_path=/chegg-prod-datalake/taxonomy_subject/mappings/bisac-tutors/subject_maps.json
2016-06-04 16:45:29,343 boto [DEBUG]:Method: HEAD
2016-06-04 16:45:29,343 boto [DEBUG]:Path: /taxonomy_subject/mappings/bisac-tutors/subject_maps.json
2016-06-04 16:45:29,343 boto [DEBUG]:Data:
2016-06-04 16:45:29,343 boto [DEBUG]:Headers: {}
2016-06-04 16:45:29,343 boto [DEBUG]:Host: chegg-prod-datalake.s3.amazonaws.com
2016-06-04 16:45:29,344 boto [DEBUG]:Port: 443
2016-06-04 16:45:29,344 boto [DEBUG]:Params: {}
2016-06-04 16:45:29,344 boto [DEBUG]:Token: None
2016-06-04 16:45:29,345 boto [DEBUG]:StringToSign:
HEAD
Sat, 04 Jun 2016 23:45:29 GMT
/chegg-prod-datalake/taxonomy_subject/mappings/bisac-tutors/subject_maps.json
2016-06-04 16:45:29,345 boto [DEBUG]:Signature:
AWS <aws_access_key>:woiRwRZEnMnzdnyZsnDilVuXSSI=
2016-06-04 16:45:29,345 boto [DEBUG]:Final headers: {'Date': 'Sat, 04 Jun 2016 23:45:29 GMT', 'Content-Length': '0', 'Authorization': u'AWS <aws_access_key>:woiRwRZEnMnzdnyZsnDilVuXSSI=', 'User-Agent': 'Boto/2.37.0 Python/2.7.10 Darwin/14.5.0'}
2016-06-04 16:45:29,395 boto [DEBUG]:Response headers: [('content-length', '1625839'), ('x-amz-id-2', 'K20p9HZyWwLWAJH/O/DSdMEhbMWgwIEfHcFyURBBB/+pgUxbS+jgvL6WLlpFVde0iPOqfZYb41U='), ('accept-ranges', 'bytes'), ('server', 'AmazonS3'), ('last-modified', 'Tue, 17 May 2016 23:50:40 GMT'), ('x-amz-request-id', 'EB94951FE2AC628A'), ('etag', '"87c8e0bef5410dd299d3427da8f2db5a"'), ('date', 'Sat, 04 Jun 2016 23:45:30 GMT'), ('x-amz-version-id', 'null'), ('content-type', 'application/json')]
----- getting contents
2016-06-04 16:45:29,396 boto [DEBUG]:path=/taxonomy_subject/mappings/bisac-tutors/subject_maps.json
2016-06-04 16:45:29,396 boto [DEBUG]:auth_path=/chegg-prod-datalake/taxonomy_subject/mappings/bisac-tutors/subject_maps.json
2016-06-04 16:45:29,396 boto [DEBUG]:path=/taxonomy_subject/mappings/bisac-tutors/subject_maps.json?versionId=null
2016-06-04 16:45:29,396 boto [DEBUG]:auth_path=/chegg-prod-datalake/taxonomy_subject/mappings/bisac-tutors/subject_maps.json?versionId=null
2016-06-04 16:45:29,396 boto [DEBUG]:Method: GET
2016-06-04 16:45:29,397 boto [DEBUG]:Path: /taxonomy_subject/mappings/bisac-tutors/subject_maps.json?versionId=null
2016-06-04 16:45:29,397 boto [DEBUG]:Data:
2016-06-04 16:45:29,397 boto [DEBUG]:Headers: {}
2016-06-04 16:45:29,397 boto [DEBUG]:Host: chegg-prod-datalake.s3.amazonaws.com
2016-06-04 16:45:29,397 boto [DEBUG]:Port: 443
2016-06-04 16:45:29,397 boto [DEBUG]:Params: {}
2016-06-04 16:45:29,397 boto [DEBUG]:Token: None
2016-06-04 16:45:29,397 boto [DEBUG]:StringToSign:
GET
Sat, 04 Jun 2016 23:45:29 GMT
/chegg-prod-datalake/taxonomy_subject/mappings/bisac-tutors/subject_maps.json?versionId=null
2016-06-04 16:45:29,398 boto [DEBUG]:Signature:
AWS <aws_access_key>:uQMF1na0f5VfbbbBkYHrNmCoWrs=
2016-06-04 16:45:29,398 boto [DEBUG]:Final headers: {'Date': 'Sat, 04 Jun 2016 23:45:29 GMT', 'Content-Length': '0', 'Authorization': u'AWS <aws_access_key>:uQMF1na0f5VfbbbBkYHrNmCoWrs=', 'User-Agent': 'Boto/2.37.0 Python/2.7.10 Darwin/14.5.0'}
2016-06-04 16:45:29,445 boto [DEBUG]:Response headers: [('x-amz-id-2', 'FHJniz3VizfJTOudCXCXXwy5vwsM4afmsll3LTir1jeOltjxaNyeCjsf0BwlbLW+eERO3dCd/iU='), ('server', 'AmazonS3'), ('transfer-encoding', 'chunked'), ('x-amz-request-id', '9823250C8F90C474'), ('date', 'Sat, 04 Jun 2016 23:45:29 GMT'), ('content-type', 'application/xml')]
Traceback (most recent call last):
File "/Users/jric/dev/common/pylib/testS3.py", line 485, in <module>
main()
File "/Users/jric/dev/common/pylib/testS3.py", line 481, in main
key.get_contents_to_filename('subject_maps.json')
File "/Library/Python/2.7/site-packages/boto/s3/key.py", line 1712, in get_contents_to_filename
response_headers=response_headers)
File "/Library/Python/2.7/site-packages/boto/s3/key.py", line 1650, in get_contents_to_file
response_headers=response_headers)
File "/Library/Python/2.7/site-packages/boto/s3/key.py", line 1482, in get_file
query_args=None)
File "/Library/Python/2.7/site-packages/boto/s3/key.py", line 1514, in _get_file_internal
override_num_retries=override_num_retries)
File "/Library/Python/2.7/site-packages/boto/s3/key.py", line 343, in open
override_num_retries=override_num_retries)
File "/Library/Python/2.7/site-packages/boto/s3/key.py", line 303, in open_read
self.resp.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>9823250C8F90C474</RequestId><HostId>FHJniz3VizfJTOudCXCXXwy5vwsM4afmsll3LTir1jeOltjxaNyeCjsf0BwlbLW+eERO3dCd/iU=</HostId></Error>