我在Python Web环境中工作，我只需使用boto的key.set_contents_from_filename（path / to / file）将文件从文件系统上传到S3。但是，我想上传一张已在网络上的图片（例如https://pbs.twimg.com/media/A9h_htACIAAaCf6.jpg:large）。
我应该以某种方式将图像下载到文件系统，然后像往常一样使用boto将其上传到S3，然后删除图像？
理想的是，如果有办法获取boto的key.set_contents_from_file或其他一些接受URL的命令，并将图像很好地传输到S3，而不必明确地将文件副本下载到我的服务器。
def upload(url):
try:
conn = boto.connect_s3(settings.AWS_ACCESS_KEY_ID, settings.AWS_SECRET_ACCESS_KEY)
bucket_name = settings.AWS_STORAGE_BUCKET_NAME
bucket = conn.get_bucket(bucket_name)
k = Key(bucket)
k.key = "test"
k.set_contents_from_file(url)
k.make_public()
return "Success?"
except Exception, e:
return e
使用set_contents_from_file，如上所述，我得到一个“字符串对象没有属性'tell'”错误。将set_contents_from_filename与url一起使用，我得到No No file或目录错误。 boto storage documentation不会上传本地文件，也不会提及上传远程存储的文件。
答案 0 :(得分：21)
好的，来自@garnaat，听起来不像S3目前允许通过网址上传。我设法通过仅将内容读入内存来将远程图像上传到S3。这很有效。
def upload(url):
try:
conn = boto.connect_s3(settings.AWS_ACCESS_KEY_ID, settings.AWS_SECRET_ACCESS_KEY)
bucket_name = settings.AWS_STORAGE_BUCKET_NAME
bucket = conn.get_bucket(bucket_name)
k = Key(bucket)
k.key = url.split('/')[::-1][0] # In my situation, ids at the end are unique
file_object = urllib2.urlopen(url) # 'Like' a file object
fp = StringIO.StringIO(file_object.read()) # Wrap object
k.set_contents_from_file(fp)
return "Success"
except Exception, e:
return e
还要感谢How can I create a GzipFile instance from the “file-like object” that urllib.urlopen() returns?
答案 1 :(得分：10)
对于这个问题的2017年相关答案，使用官方的'boto3'包（而不是原始答案中的旧'boto'包）：
Python 3.5
如果您正在进行干净的Python安装，请先pip安装两个软件包：
pip install boto3
pip install requests
import boto3
import requests
# Uses the creds in ~/.aws/credentials
s3 = boto3.resource('s3')
bucket_name_to_upload_image_to = 'photos'
s3_image_filename = 'test_s3_image.png'
internet_image_url = 'https://docs.python.org/3.7/_static/py.png'
# Do this as a quick and easy check to make sure your S3 access is OK
for bucket in s3.buckets.all():
if bucket.name == bucket_name_to_upload_image_to:
print('Good to go. Found the bucket to upload the image into.')
good_to_go = True
if not good_to_go:
print('Not seeing your s3 bucket, might want to double check permissions in IAM')
# Given an Internet-accessible URL, download the image and upload it to S3,
# without needing to persist the image to disk locally
req_for_image = requests.get(internet_image_url, stream=True)
file_object_from_req = req_for_image.raw
req_data = file_object_from_req.read()
# Do the actual upload to s3
s3.Bucket(bucket_name_to_upload_image_to).put_object(Key=s3_image_filename, Body=req_data)
答案 2 :(得分：7)
不幸的是，确实没有办法做到这一点。至少现在不是。我们可以向boto添加一个方法，比如set_contents_from_url，但该方法仍然需要将文件下载到本地计算机然后上传它。它可能仍然是一种方便的方法，但它不会为您节省任何费用。
为了做你真正想做的事情，我们需要在S3服务本身上有一些功能，允许我们传递URL并让它为我们存储URL。这听起来像一个非常有用的功能。您可能希望将其发布到S3论坛。
答案 3 :(得分：5)
这是我使用requests的方式，关键是在最初发出请求时设置stream=True，然后使用upload.fileobj()方法上传到s3：
import requests
import boto3
url = "https://upload.wikimedia.org/wikipedia/en/a/a9/Example.jpg"
r = requests.get(url, stream=True)
session = boto3.Session()
s3 = session.resource('s3')
bucket_name = 'your-bucket-name'
key = 'your-key-name'
bucket = s3.Bucket(bucket_name)
bucket.upload_fileobj(r.raw, key_name)
答案 4 :(得分：1)
使用boto3 upload_fileobj方法，您可以将文件流式传输到S3存储桶，而无需保存到磁盘。这是我的功能：
import boto3
import StringIO
import contextlib
import requests
def upload(url):
# Get the service client
s3 = boto3.client('s3')
# Rember to se stream = True.
with contextlib.closing(requests.get(url, stream=True, verify=False)) as response:
# Set up file stream from response content.
fp = StringIO.StringIO(response.content)
# Upload data to S3
s3.upload_fileobj(fp, 'my-bucket', 'my-dir/' + url.split('/')[-1])
答案 5 :(得分：1)
一个简单的3行实现，可直接使用lambda：
import boto3
import requests
s3_object = boto3.resource('s3').Object(bucket_name, object_key)
with requests.get(url, stream=True) as r:
s3_object.put(Body=r.content)
.get部分的来源直接来自requests documentation
答案 6 :(得分：0)
import boto
from boto.s3.key import Key
from boto.s3.connection import OrdinaryCallingFormat
from urllib import urlopen
def upload_images_s3(img_url):
try:
connection = boto.connect_s3('access_key', 'secret_key', calling_format=OrdinaryCallingFormat())
bucket = connection.get_bucket('boto-demo-1519388451')
file_obj = Key(bucket)
file_obj.key = img_url.split('/')[::-1][0]
fp = urlopen(img_url)
result = file_obj.set_contents_from_string(fp.read())
except Exception, e:
return e
答案 7 :(得分：0)
S3目前似乎不支持远程上传。您可以使用下面的类将图像上传到S3。此处的上载方法首先尝试下载图像并将其保留在内存中一段时间，直到被上载。为了能够连接到S3，您将必须使用命令pip install awscli安装AWS CLI，然后使用命令aws configure输入少量凭证：
import urllib3
import uuid
from pathlib import Path
from io import BytesIO
from errors import custom_exceptions as cex
BUCKET_NAME = "xxx.yyy.zzz"
POSTERS_BASE_PATH = "assets/wallcontent"
CLOUDFRONT_BASE_URL = "https://xxx.cloudfront.net/"
class S3(object):
def __init__(self):
self.client = boto3.client('s3')
self.bucket_name = BUCKET_NAME
self.posters_base_path = POSTERS_BASE_PATH
def __download_image(self, url):
manager = urllib3.PoolManager()
try:
res = manager.request('GET', url)
except Exception:
print("Could not download the image from URL: ", url)
raise cex.ImageDownloadFailed
return BytesIO(res.data) # any file-like object that implements read()
def upload_image(self, url):
try:
image_file = self.__download_image(url)
except cex.ImageDownloadFailed:
raise cex.ImageUploadFailed
extension = Path(url).suffix
id = uuid.uuid1().hex + extension
final_path = self.posters_base_path + "/" + id
try:
self.client.upload_fileobj(image_file,
self.bucket_name,
final_path
)
except Exception:
print("Image Upload Error for URL: ", url)
raise cex.ImageUploadFailed
return CLOUDFRONT_BASE_URL + id
答案 8 :(得分：0)
from io import BytesIO
def send_image_to_s3(url, name):
print("sending image")
bucket_name = 'XXX'
AWS_SECRET_ACCESS_KEY = "XXX"
AWS_ACCESS_KEY_ID = "XXX"
s3 = boto3.client('s3', aws_access_key_id=AWS_ACCESS_KEY_ID,
aws_secret_access_key=AWS_SECRET_ACCESS_KEY)
response = requests.get(url)
img = BytesIO(response.content)
file_name = f'path/{name}'
print('sending {}'.format(file_name))
r = s3.upload_fileobj(img, bucket_name, file_name)
s3_path = 'path/' + name
return s3_path