S3QL is a filesystem that stores all its data in a remote storage system, such as Orbit. It can only be mounted on one server at a time so it’s unsuitable for load-balanced clusters but it provides an easy way to add highly-available and auto-scalable storage to a single server.
This guide will take you through installing and configuring an Orbit-backed filesystem on an Ubuntu cloud server, suitable for storing web application assets (such as Wordpress uploads). Files written to the filesystem are cached locally and auto-synced up to Orbit. In the event of a server failure or even datacentre outage, the filesystem can be remounted on another server in another zone.
We’ll be using API client credentials to authenticate with Orbit, so firstly,
create an API client
using the Brightbox Manager GUI. Make sure to set its privileges to
Storage Only and note the identifier and secret.
Then, SSH into your server and install S3QL:
$ sudo apt-get install -y s3ql
$ sudo install -d -m 0755 /srv/share $ sudo install -d -m 0700 /var/cache/s3ql
The export directory will be where S3QL caches data for fast local access - we’ll tell S3QL to limit the size of the cache, don’t worry. The
/srv/share directory is where we’ll mount the filesystem.
Create a file named
In the file, specify the Orbit url, your container name and the API client identifier and secret:
[default] storage-url: swift://orbit.brightbox.com/myfilesystem backend-login: cli-aaaaa backend-password: secret
And set the file permissions so only root can read it:
sudo chmod 0600 /etc/s3ql.authinfo sudo chown root.root /etc/s3ql.authinfo
Now we have to initialise the filesystem using the
$ sudo mkfs.s3ql --cachedir /var/cache/s3ql --authfile /etc/s3ql.authinfo -L myfilesystem --plain swift://orbit.brightbox.com/myfilesystem Creating metadata tables... Dumping metadata... Dumping metadata... Compressing and uploading metadata... Wrote 111 bytes of compressed metadata. Cycling metadata backups... Backing up old metadata...
Before going any further, let’s test mounting the filesystem:
$ sudo mount.s3ql --cachedir /var/cache/s3ql --authfile /etc/s3ql.authinfo --allow-other swift://orbit.brightbox.com/myfilesystem /srv/share/ Using 4 upload threads. Autodetected 65492 file descriptors available for cache entries Using cached metadata. Setting cache size to 30979 MB Mounting filesystem... $ df -h | grep share swift://orbit.brightbox.com/myfilesystem/ 1.0T 0 1.0T 0% /srv/share $ sudo umount /srv/share
Now we just configure S3QL to start on boot and mount the filesystem.
If you’re using Xenial, you need to write a systemd config file:
Create a file named
/lib/systemd/system/s3ql.service with the contents:
[Unit] Description=mount s3ql filesystem Wants=network-online.target [Service] ExecStart=/usr/bin/mount.s3ql --fg --cachedir /var/cache/s3ql --authfile /etc/s3ql.authinfo --compress none --cachesize 1048576 --allow-other swift://orbit.brightbox.com/myfilesystem /srv/share/ ExecStop=/usr/bin/umount.s3ql /srv/share TimeoutStopSec=42
Once configured, enable the service and start it:
$ sudo systemctl enable s3ql Created symlink from /etc/systemd/system/multi-user.target.wants/s3ql.service to /lib/systemd/system/s3ql.service. $ sudo systemctl start s3ql
If you’re using Trusty, the instead of systemd you need to write an upstart config:
Create a file names
/etc/init/s3ql.conf with the contents:
start on (started networking) stop on starting rc RUNLEVEL= expect stop kill timeout 300 limit nofile 66000 66000 console log pre-stop script umount.s3ql /srv/share end script exec /usr/bin/mount.s3ql --upstart --cachedir /var/cache/s3ql --authfile /etc/s3ql.authinfo --compress none --cachesize 1048576 --allow-other swift://orbit.brightbox.com/myfilesystem /srv/share/
Then start the service:
$ sudo start s3ql
We’re disabling compression here with the
--compress none options because it’s
just wasting cpu if the files you’ll be storing aren’t compressible - you can
leave that option out if you want compression enabled as per the default.
We’re also setting
cachesize to 1GB. The larger the cache size, the less data
s3ql will have to pull from Orbit on reads, so the better the overall
performance. It’s a good idea to make this at least as big as your working set
(the set of files that are most commonly in use) so they aren’t repeatedly
downloaded from Orbit.
Also, the cache size should ideally be at least ten times the size of the biggest file you’ll be storing on the filesystem, so 1GB accommodates up to 100M files well. You can still store larger files, but the performance can be very poor.
As with any filesystem, just use
chmod as usual to grant access to
files and directories for non-root users.
If your server is rebooted without cleanly unmounting the S3QL filesystem for
some reason, you’ll have to use the
fsck.s3ql command to validate and fixup
the s3ql filesystem:
$ sudo fsck.s3ql --cachedir /var/cache/s3ql --authfile /etc/s3ql.authinfo swift://orbit.brightbox.com/myfilesystem
S3QL supports copy-on-write snapshots, handy for fast efficient backups. Say
your data is in a directory named
/srv/share/data, you take a snapshot of it
s3qlcp command and store it in your snapshots directory:
$ sudo du -sh /srv/share/data/ 78M /srv/share/data/ $ sudo mkdir /srv/share/snapshots $ sudo s3qlcp /srv/share/data /srv/share/snapshots/data-20160830 $ sudo du -sh /srv/share/snapshots/data-20160830/ 78M /srv/share/snapshots/data-20160830/
The resulting snapshot is almost instantaneous and takes no extra space (until
files in the original data directory are modified). You can also make the
resulting snapshot directory immutable, to prevent it being removed or modified
itself, using the
$ sudo s3qllock /srv/share/snapshots/data-20160830/
If you later need to delete your snapshot, you can remove it using the
$!shell $ sudo s3qlrm /srv/share/snapshots/data-20160830/
You can use the
s3qlstat command to get information about a mounted S3QL
filesystem. You can see that this one with the snaphot I just made is listed as
~154 MiB of data despite only actually using 71 MiB, thanks to the copy-on-write:
$ sudo s3qlstat /srv/share/ Directory entries: 1703 Inodes: 1725 Data blocks: 442 Total data size: 154 MiB After de-duplication: 71.0 MiB (46.01% of total) After compression: 71.0 MiB (46.01% of total, 100.00% of de-duplicated) Database size: 320 KiB (uncompressed) Cache usage: 77.1 MiB (dirty: 0 bytes)
Data is already encrypted by Orbit in transit and at rest, but you can do your own encrypting within S3QL prior to uploading to Orbit if you prefer. See the S3QL documentation for more details.
Since S3QL maintains its own local database of metadata most normal filesystem actions (listing files, creating directories etc) are almost as fast as using a native filesystem. Reading data will be limited to the speed at which the data can be downloaded from Orbit, unless it’s in the cache in which case, again, it’s almost native speeds. Writes are first buffered to the cache, so again are very fast but if S3QL is busy processing and uploading data, it can slow down a bit.
So it’s great for handling uploaded assets from your web application but it’s not recommend for storing MySQL databases or anything like that.
Last updated: 01 Sep 2016 at 15:32 UTC