在Elastic Beanstalk上使用boto3和django-storage将文件上传到S3时出现500错误

时间:2018-02-08 06:45:18

标签: django python-3.x amazon-s3 boto3 django-filer

尝试通过部署到Elastic Beanstalk的应用程序中的django-cms Filer管理员上传文件时,我收到以下500 http错误:

在浏览器控制台中,来自fileuploader.min.js:26

POST http://app-staging.us-east-1.elasticbeanstalk.com/admin/filer/clipboard/operations/upload/1/?qqfile=example.jpg 500 (Internal Server Error)

我正在使用django-cms(因此easy-thumbnails),django-storagesboto3。部署时,collectstatic配置文件中的.ebextensions命令运行良好,文件都很好地存储在/staging/static/的S3存储桶中。

删除DEFAULT_FILE_STORAGE赋值可以正常工作,但显然是站不住脚的,因为对beanstalk的任何更改的每个eb deploy都会导致新的实例并删除以前的文件上传。

堆栈追踪:

Internal Server Error: /admin/filer/clipboard/operations/upload/1/
 Traceback (most recent call last):
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
     response = get_response(request)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
     response = self._get_response(request)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
     response = self.process_exception_by_middleware(e, request)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
     response = wrapped_callback(request, *callback_args, **callback_kwargs)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
     return view_func(*args, **kwargs)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/filer/admin/clipboardadmin.py", line 114, in ajax_upload
     file_obj.save()
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/filer/models/imagemodels.py", line 53, in save
     super(Image, self).save(*args, **kwargs)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/filer/models/abstract.py", line 74, in save
     super(BaseImage, self).save(*args, **kwargs)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/filer/models/filemodels.py", line 194, in save
     super(File, self).save(*args, **kwargs)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/polymorphic/models.py", line 74, in save
     return super(PolymorphicModel, self).save(*args, **kwargs)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/models/base.py", line 806, in save
     force_update=force_update, update_fields=update_fields)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/models/base.py", line 835, in save_base
     self._save_parents(cls, using, update_fields)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/models/base.py", line 862, in _save_parents
     self._save_table(cls=parent, using=using, update_fields=update_fields)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/models/base.py", line 922, in _save_table
     result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/models/base.py", line 961, in _do_insert
     using=using, raw=raw)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/models/manager.py", line 85, in manager_method
     return getattr(self.get_queryset(), name)(*args, **kwargs)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/models/query.py", line 1060, in _insert
     return query.get_compiler(using=using).execute_sql(return_id)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
     for sql, params in self.as_sql():
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1051, in as_sql
     for obj in self.query.objs
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1051, in <listcomp>
     for obj in self.query.objs
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1050, in <listcomp>
     [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1000, in pre_save_val
     return field.pre_save(obj, add=True)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/models/fields/files.py", line 297, in pre_save
     file.save(file.name, file.file, save=False)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/filer/fields/multistorage_file.py", line 122, in save
     super(MultiStorageFieldFile, self).save(name, content, save)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/easy_thumbnails/files.py", line 670, in save
     super(ThumbnailerFieldFile, self).save(name, content, *args, **kwargs)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/models/fields/files.py", line 95, in save
     self.name = self.storage.save(name, content, max_length=self.field.max_length)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/core/files/storage.py", line 54, in save
     return self._save(name, content)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 452, in _save
     self._save_content(obj, content, parameters=parameters)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 467, in _save_content
     obj.upload_fileobj(content, ExtraArgs=put_parameters)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/boto3/s3/inject.py", line 513, in object_upload_fileobj
     ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/boto3/s3/inject.py", line 431, in upload_fileobj
     return future.result()
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/s3transfer/futures.py", line 73, in result
     return self._coordinator.result()
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/s3transfer/futures.py", line 233, in result
     raise self._exception
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/s3transfer/tasks.py", line 126, in __call__
     return self._execute_main(kwargs)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/s3transfer/tasks.py", line 150, in _execute_main
     return_value = self._main(**kwargs)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/s3transfer/upload.py", line 685, in _main
     client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/client.py", line 317, in _api_call
     return self._make_api_call(operation_name, kwargs)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/client.py", line 589, in _make_api_call
     api_params, operation_model, context=request_context)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/client.py", line 624, in _convert_to_request_dict
     api_params, operation_model)
   File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/validate.py", line 291, in serialize_to_request
     raise ParamValidationError(report=report.generate_report())
 botocore.exceptions.ParamValidationError: Parameter validation failed:
 Invalid type for parameter ContentType, value: None, type: <class 'NoneType'>, valid types: <class 'str'>

settings.py

STATIC_ROOT = os.path.join(BASE_DIR, '..', 'www', 'static')
STATICFILES_DIRS = (
    os.path.join(BASE_DIR, 'static', 'eatingveg'),
)

MEDIA_ROOT = os.path.join(BASE_DIR, '..', 'www', 'media')

# AWS S3 Settings
AWS_STORAGE_BUCKET_NAME = os.environ.get('EV_S3_BUCKET_NAME')
AWS_S3_REGION_NAME = os.environ.get('EV_S3_REGION_NAME')  # e.g. us-east-2
AWS_ACCESS_KEY_ID = os.environ.get('EV_AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('EV_AWS_SECRET')
AWS_QUERYSTRING_AUTH = False
AWS_IS_GZIPPED = True

# Tell django-storages the domain to use to refer to static files.
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
STATIC_URL = "https://%s/staging/" % AWS_S3_CUSTOM_DOMAIN
MEDIA_URL = "https://%s/staging/media/" % AWS_S3_CUSTOM_DOMAIN

# Tell the staticfiles app to use S3Boto3 (via our StaticStorage subclass)
# storage when writing the collected static files
# (when you run `collectstatic`).
STATICFILES_LOCATION = 'staging/static'
STATICFILES_STORAGE = 'custom_storages.StaticStorage'

MEDIAFILES_LOCATION = 'staging/media'
DEFAULT_FILE_STORAGE = 'custom_storages.MediaStorage'

custom_storages.py

# custom_storages.py
from django.conf import settings
from storages.backends.s3boto3 import S3Boto3Storage


class StaticStorage(S3Boto3Storage):
    location = settings.STATICFILES_LOCATION


class MediaStorage(S3Boto3Storage):
    location = settings.MEDIAFILES_LOCATION

2 个答案:

答案 0 :(得分:0)

事实证明问题出在django-storages。通过Filer管理员添加文件时,它正在调用boto,其中boto不喜欢空ContentType,如文档here所示。通过允许None来修复此问题pull request已提交,但尚未合并。

通过分配bxm156的PR,我能够解决问题。

答案 1 :(得分:0)

将django-storages更新为1.6.6。它解决了这个问题。