Amazon S3 – What Is It & What Does It Do?

When most people use S3 (Simple Storage Service), the first thing they think about is Amazon Cloud Storage. However, the S3 protocol is actually used in many places outside of AWS. S3 is used for interfacing with object storage over a network, by using buckets, keys, and operations. One example is how Ceph implements it via its RADOS Gateways (RGW). While it’s true that the S3 API (Application Programming Interface) is developed and released by Amazon, it has implementations within a wide variety of storage systems. The level of complexity for these solutions vary; from basic CRUD (create, remove, update, delete) functions, to full S3 Compatibility. The goal of this blog is to outline some of the most common implementations of the S3 protocol. 

S3 is what is considered an HTTP REST API. It’s an API that uses HTTP requests to get, put, post and delete data. The REST API is considered a “stateless” protocol, where the server does not store any state about the client sessions on its side. This means that each request from the client has to contain all the information necessary to understand the request, and doesn’t have the ability to take advantage of any stored context on the server-side – the client is responsible for handling application state information.

S3 at 45Dives

Clustering

At 45 Drives, we use Ceph for our clustered storage solutions, which has its underlying architecture built on object storage. Ceph offers what is called RADOS Gateway (or RGW). This is an object storage interface, which is built on top of the librados API to provide applications with a RESTfulgateway into the cluster. RGW currently has two supported interfaces:

1. S3-Compatible: This provides object storage functionality, with an interface that is compatible with most of the Amazon S3 API

2. Swift-compatible: This provides object storage functionality with an interface that is compatible with most of the OpenStack Swift API.

Single Server Solutions

FreeNAS

FreeNAS is the most used single-server operating system both at 45Drives and in the entire industry. FreeNAS uses ZFS as its underlying file system and logical volume manager. FreeNAS has great built-in functionality to connect to many different cloud deployments, including S3. Once a user connects via their S3 credentials, it allows them to replicate to the chosen S3 buckets at any created schedule.

CentOS, and some other commonly used storage operating systems have native tools as well. If you have questions around configuring your Storinator to access or sync with S3 buckets, your best option is to contact our account managers.

Interfacing Options

Cyberduck

Cyberduck is an open-source client that allows many different protocols, such as FTP and WebDAV. It also supports cloud storage such as Swift, Amazon S3, Backblaze, and Azure. For S3, Cyberduck allows you to browse directly into S3 buckets where you can add and remove files. Cyberduck works on Windows and Mac. It allows you to interface with S3 through a simple, great looking client and provides most of the basic functionality you need.

S3 browser

This is an extremely powerful tool to browse and work inside S3 buckets via a downloadable client for Windows. It allows many extended features that Cyberduck does not; such as changing permissions, multiple users, ACLs, previewing certain file types from within the S3 browser itself and many more. Honestly, it isn’t the best-looking option. Feature-wise however, it is one of the most robust solutions available.

S3FS

This is a Linux userspace file-system application, that allows you to mount an S3 bucket as a local file system on your machine. S3FS can operate in 2 different modes: mount mode and command mode. In command mode, it allows the user to manipulate the s3 buckets in various ways. S3FS can be incredibly useful for Linux production environments that are looking to back up their active datasets to the cloud using RSYNC with an S3 bucket. S3FS provides an easy and smooth way to integrate a Linux desktop in with your S3 bucket.

Nextcloud

Nextcloud is free and open-source software that is built to create your own file hosting service. Nextcloud aims to be a free, do-it-yourself, all-in-one Dropbox replacement. The web application allows you to open and view files from within the web interface, use its built-in Office software suite and much more. One great feature of Nextcloud is its External Storage add-on. This allows you to directly mount an S3 bucket to your Nextcloud server. From there, any user who you give access to will be able to access any files in your S3 bucket.

So, let’s wrap it up like this, think of S3 as the English language (Amazon S3), even though there are 195 of countries in the world using their own native language (Ceph object store), many people still communicate using English.