我有一个应该连接到S3的脚本,找到一个特定的存储桶,然后为该存储桶中的每个新密钥获取get_contents_to_filename
conn = S3Connection(ACCESS_KEY_ID, SECRET_ACCESS_KEY, debug=1,calling_format=ProtocolIndependentOrdinaryCallingFormat())
bucket = Bucket(conn, 'bucket_name_here')
这是获取存储桶中密钥的循环
# Loop through each key for each hour of the day
for idx, key in enumerate(bucket.list(PATH)):
logger2.info('Key: %s' % str(key))
# Create new filename for output - appid_yyyy_mm_dd_hh
file_name = app_id + dt.strftime("%Y_%m_%d") + "_" + str(idx) + '.log.gz'
current_file = dateutil.parser.parse(key.last_modified[:19])
# print current time and last observer time
logger3.info('Current File Datetime:'+ ' ' + str(current_file) + ' ' +
'Last Modified Datetime:' + ' ' + str(max_file_datetime))
if current_file <= max_file_datetime:
logger.info('skipping file: ' + str(key.name))
else:
try:
logger.info('downloading file' + str(key.name))
key.get_contents_to_filename(outlocation + file_name,cb=mycb,num_cb=5)
except Exception, e:
logger.error('Unable to download Key contents to file: %s' %str(e))
我从我公司的代理服务器后面运行此代码 - 目前使用cntlm尝试绕过它但由于连接问题我在get_contents_to_filename上一直收到错误
这是我得到的错误:
socket.error:[Errno 10060]连接尝试失败,因为连接方没有正确响应 由于连接的主机未能响应,因此时间点或建立的连接失败
当我打开调试时,这是输出:
2015-07-30 19:13:52,427 foo [DEBUG]:Using access key provided by
client.
2015-07-30 19:13:52,427 foo [DEBUG]:Using secret key provided by client.
2015-07-30 19:13:52,489 foo [DEBUG]:path=/------string------/
2015-07-30 19:13:52,489 foo [DEBUG]:auth_path=/------string------/
2015-07-30 19:13:52,489 foo [DEBUG]:path=/------string------/?prefix-------------------------
2015-07-30 19:13:52,489 foo [DEBUG]:auth_path=/------string------/?prefix-------------------------
2015-07-30 19:13:52,489 foo [DEBUG]:Method: GET
2015-07-30 19:13:52,489 foo [DEBUG]:Path: /------string------/?prefix-------------------------
2015-07-30 19:13:52,489 foo [DEBUG]:Data:
2015-07-30 19:13:52,489 foo [DEBUG]:Headers: {}
2015-07-30 19:13:52,489 foo [DEBUG]:Host: s3.amazonaws.com
2015-07-30 19:13:52,489 foo [DEBUG]:Port: 443
2015-07-30 19:13:52,489 foo [DEBUG]:Params: {}
2015-07-30 19:13:52,489 foo [DEBUG]:establishing HTTPS connection: host=s3.amazonaws.com, kwargs={'port': 443, 'timeout'
: 70}
2015-07-30 19:13:52,941 foo [DEBUG]:Token: None
2015-07-30 19:13:52,941 foo [DEBUG]:StringToSign:
GET
Thu, 30 Jul 2015 18:13:52 GMT
/exports.localytics/
2015-07-30 19:13:52,941 foo [DEBUG]:Signature:
AWS ---------------string---------------=
2015-07-30 19:13:52,941 foo [DEBUG]:Final headers: {'Date': 'Thu, 30 Jul 2015 18:13:52 GMT', 'Content-Length': '0', 'Aut
horization': u'-----string_here----------', 'User-Agent': 'Boto/2.38.0 Python/2.7.10 Windows/
8'}
2015-07-30 19:14:13,999 foo [DEBUG]:encountered error exception, reconnecting
2015-07-30 19:14:13,999 foo [DEBUG]:establishing HTTPS connection: host=s3.amazonaws.com, kwargs={'port': 443, 'timeout'
: 70}
2015-07-30 19:14:14,171 foo [DEBUG]:Token: None
2015-07-30 19:14:14,171 foo [DEBUG]:StringToSign:
GET
调试输出的最后一部分在连接重试时重复几次,然后我得到socket.error。我对boto / s3的所有内容都很陌生,并且对python很新,所以对任何不良做法表示歉意。
是代理问题还是其他问题?