Highly available filesystem with S3FS and Orbit

S3FS is a filesystem for Linux that stores all its data in S3 compatible systems such as Brightbox Orbit. It provides an easy way to add highly-available and auto-scalable storage to a single server.

This guide will take you through installing and configuring an Orbit-backed filesystem on a Ubuntu cloud server, suitable for storing any kind of asset (but particularly web application uploads). Files are written directly to the filesystem stored in Orbit.

We’re assuming that you’ve already signed up with Brightbox, have provided your public SSH key and have built a Ubuntu Xenial server.

Create an API client and Orbit container

We’ll be using API client credentials to authenticate with Orbit, so firstly, create an API client using the Brightbox Manager GUI. Make sure to set its privileges to Orbit Storage Only and note the identifier and secret.

And then create an Orbit container named as you’d like (we’ll use mys3filesystem in this example) and give this API client read and write permissions to it.

Set your credentials

First, SSH into your server and configure your API client identifier and secret into the s3fs password file:

$ echo "cli-abcd1:secret-key-here" | sudo tee /etc/s3fs.auth
$ sudo chmod 600 /etc/s3fs.auth

Install S3FS

Then install S3FS:

$ sudo apt-get install -y s3fs

Make directories for the mount point

In this guide, the Orbit container will be mounted in /srv/share but you could choose any empty directory.

$ sudo install -d -m 0755 /srv/share

Mount your Orbit container with S3FS

Use S3FS to mount your Orbit container at the mount point you created above, replacing the arguments as necessary:

$ sudo s3fs mys3filesystem /srv/share -o passwd_file=~/etc/s3fs.auth -o url=https://orbit.brightbox.com/ -o endpoint=gb1 -o use_path_request_style

Your orbit container is now connected!

Test the filesystem

Let’s check it’s all working.

Create a test file:

$ echo -e "Hello World" | sudo tee /srv/share/test.txt

Unmount the filesystem:

$ sudo umount /srv/share

Check the folder is empty:

$ ls /srv/share

Remount the folder:

$ sudo s3fs mys3filesystem /srv/share -o passwd_file=~/etc/s3fs.auth -o url=https://orbit.brightbox.com/ -o endpoint=gb1 -o use_path_request_style

Check the folder is contains text.txt:

$ ls /srv/share

Check the contents of test.txt

$ cat /srv/share/text.txt

Configure it to start on boot

We often will want this to survive a reboot, so we will want to configure S3FS to start on boot and mount the filesystem.

To do this on Ubuntu Xenial, we’ll use a systemd config file:

Create a file named /lib/systemd/system/s3fs.service with the contents including the mount command we used earlier:

 [Unit]
 Description=mount s3fs filesystem
 Wants=network-online.target
 
 [Service]
 ExecStart=s3fs mys3filesystem /srv/share -o passwd_file=/etc/s3fs.auth -o url=https://orbit.brightbox.com/ -o endpoint=gb1 -o use_path_request_style
 ExecStop=/usr/bin/umount /srv/share
 TimeoutStopSec=42

Once configured, enable the service and start it:

 $ sudo systemctl enable s3fs
 Created symlink from /etc/systemd/system/multi-user.target.wants/s3fs.service to /lib/systemd/system/s3fs.service.
 
 $ sudo systemctl start s3fs

Now you can try rebooting - and testing the filesystem afterwards!

File permissions

As with any filesystem, just use chown and chmod as usual to grant access to files and directories for non-root users.

Tuning and Performance

There are lots of advanced caching and tuning options available that you might consider to improve performance if that’s something you want.

S3FS is great for handling uploaded assets from your web application but it’s not recommend for storing MySQL databases or serving web applications from.

Similar tools

Another option for storing files on Orbit and accessing them through your filesystem is S3QL. Rather than using our S3-API compatibility interface, S3QL uses our faster Swift interface directly and also has an advanced caching layer. However because it doesn’t store objects in a standard way, you can’t access the objects directly via orbit, as you can with S3FS.