问题：我有一个通过多配置jenkins作业运行的python脚本。作业下载脚本并将参数传递给它。它产生了10次，因为有10组数据并行加载。在某些情况下，并不总是2或3个作业将得到s3超时错误。我不确定为什么，因为它不总是相同的工作，而不是所有的时间。
错误：
Downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/WOK5_20150410045810_WOS_0124.json.gz
Downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/WOK5_20150410045810_WOS_0125.json.gz
Downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/WOK5_20150410045810_WOS_0126.json.gz
Downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/WOK5_20150410045810_WOS_0127.json.gz
Downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/WOK5_20150410045810_WOS_0128.json.gz
Downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/WOK5_20150410045810_WOS_0129.json.gz
Downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/WOK5_20150410045810_WOS_0130.json.gz
Downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/WOK5_20150410045810_WOS_0131.json.gz
Downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/WOK5_20150410045810_WOS_0132.json.gz
Downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/WOK5_20150410045810_WOS_0133.json.gz
Downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/WOK5_20150410045810_WOS_0134.json.gz
Downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/WOK5_20150410045810_WOS_0135.json.gz
> Downloading:
> /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/WOK5_20150410045810_WOS_0135.json.gz
> Traceback (most recent call last): File
> "/var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/es_single_load.py",
> line 1094, in <module>
> directory = download_s3(wos_data, 'tr-ips-ses-data',access_key, secret_key) File
> "/var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/es_single_load.py",
> line 31, in download_s3
> l.get_contents_to_filename(d) File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 1712, in
> get_contents_to_filename
> response_headers=response_headers) File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 1650, in
> get_contents_to_file
> response_headers=response_headers) File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 1482, in
> get_file
> query_args=None) File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 1535, in
> _get_file_internal
> for bytes in self: File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 386, in next
> data = self.resp.read(self.BufferSize) File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 413, in
> read
> return http_client.HTTPResponse.read(self, amt) File "/usr/lib64/python2.7/httplib.py", line 573, in read
> s = self.fp.read(amt) File "/usr/lib64/python2.7/socket.py", line 380, in read
> data = self._sock.recv(left) File "/usr/lib64/python2.7/ssl.py", line 714, in recv
> return self.read(buflen) File "/usr/lib64/python2.7/ssl.py", line 608, in read
> v = self._sslobj.read(len or 1024) ssl.SSLError: ('The read operation timed out',) Build step 'Execute shell' marked build as
> failure Finished: FAILURE
蟒：
def download_s3(wos,bucket, access_key, secret_key):
# connect to the bucket
conn = boto.connect_s3(access_key, secret_key)
bucket = conn.get_bucket(bucket)
LOCAL_PATH = os.path.dirname(os.path.realpath(__file__))
bucket_list = bucket.list()
for l in bucket_list:
keyString = str(l.key)
d = LOCAL_PATH + '/' + keyString
search_string = '/' + wos + '/'
if search_string in str(l) and '20150707' in str(l):
if str(wos) == 'wos_1' and '/wos_10/' in str(l):
continue
try:
print 'Downloading: %s' % d
l.get_contents_to_filename(d)
except OSError:
if not os.path.exists(d):
os.makedirs(d)
else:
continue
conn.close()
return d,LOCAL_PATH