What is the best approach in nodejs to switch between the local filesystem and Amazon S3 filesystem?

Question

I'm looking for the best way to switch between using the local filesystem and the Amazon S3 filesystem.

I think ideally I would like a wrapper to both filesystems that I can code against. A configuration change would tell the wrapper which filesystem to use. This is useful to me because a developer can use their local filesystem, but our hosted environments can use Amazon S3 by just changing a configuration option.

Are there any existing wrappers that do this? Should I write my own wrapper? Is there another approach I am not aware of that would be better?

Answer 1

There's a project named s3fs that offers a subset of POSIX file system function on top of S3. There's no native Amazon-provided way to do this.

However, you should think long and hard about whether or not this is a sensible option. S3 is an object store, not a regular file system, and it has quite different performance and latency characteristics.

If you're looking for high iops, NAS-style storage then Amazon EFS (in preview) would be more appropriate. Or roll your own NFS/CIFS solution using EBS volumes or SoftNAS or Gluster.

Answer 2

I like your idea to build a wrapper that can use either the local file system or S3. I'm not aware of anything existing that would provide that for you, but would certainly be interested to hear if you find anything.

An alternative would be to use some sort of S3 file system mount, so that your application can always use standard file system I/O but the data might be written to S3 if your system has that location configured as an S3 mount. I don't recommend this approach because I've never heard of an S3 mounting solution that didn't have issues.

Another alternative is to only design your application to use S3, and then use some sort of S3 compatible local object storage in your development environment. There are several answers to this question that could provide an S3 compatible service during development.

Answer 3

有一种名为JuiceFS的服务可以满足您的需求。

根据他们的文档：

  

JuiceFS是专门设计用于POSIX的共享文件系统   在云端工作。

     

它旨在在云中运行，因此您可以利用便宜的价格   对象存储服务可以经济地存储您的数据。

     

这是一个   兼容POSIX的文件系统，因此您可以无缝访问数据   访问本地文件。

     

这是一个共享文件系统，因此您可以共享您的   多台计算机上的文件。

s3是受支持的后端之一，您甚至可以对其进行配置，以将文件复制到另一个云上的另一个对象存储系统。