Using Rclone with Orbit

This guide will take you through installing and configuring Rclone to copy files to and from Orbit object storage.

Rclone is a command line tool designed for managing files on cloud storage. It is inspired by rsync and currently has support for over 70 different cloud storage services.

Rclone includes support for Orbit through the Openstack Swift remote type.

Prerequisites

For the purposes of this guide, we’ll assume that you have already:

1. Signed up with a Brightbox account
2. Created an Orbit container and API Client

You’ll need the following details:

• Account ID (acc-xxxxx)
• Orbit container name
• API client (cli-xxxxx)
• API client secret

Install Rclone

For our guide we’re using an Ubuntu 22.04 cloud server, but you could also install Rclone on macOS or Windows too.

$ sudo apt update
$ sudo apt install rclone -y

Creating an Rclone config

If you’re using Rclone manually, you can create a config file to save having to specify the options each time.

Rclone has an interactive config tool, but many of the options aren’t relevant so it’s easier to just create the config file directly:

$ nano ~/.config/rclone/rclone.conf

Enter the following details and save the config file:

[orbit]
type = swift
tenant = <account_id>
user = <api_client_id>
key = <api_client_secret>
auth = https://orbit.brightbox.com/v3

Using Rclone without a config file

It’s worth noting here that you don’t need a config file to use Rclone, you can specify all of the options on the commandline, which is how you’d use it with scripts.

$ rclone --swift-auth https://orbit.brightbox.com/v3 \
--swift-tenant <account_id> \
--swift-user <api_client_id> \
--swift-key <api_client_secret> \
ls :swift:my-container

Note that without a config file, we need to use a feature in Rclone to create a remote on-the-fly by prefixing the remote type with a colon i.e. :swift:

Listing files

Now that we have configured rclone we should be able to access our Orbit container and list the contents:

$ rclone ls orbit:my-container

Uploading files

To upload a file to our Orbit container we use the rclone copy command:

$ rclone copy archive.tar.gz orbit:my-container

Downloading files

To download a file from an Orbit container:

$ rclone copy orbit:my-container/archive.tar.gz ~/my-local-files/

Deleting files

To delete files we can use rclone delete:

$ rclone delete orbit:my-container/archive.tar.gz

The delete command also provides a number of useful include/exclude filters.

To blindly delete a path and all of its contents without filtering, we can use the purge command:

$ rclone delete orbit:my-container/old_backups/

Large file support

Rclone includes support for uploading large files to Orbit using Openstack Swift Large Object Support.

Rclone uses a default chunk size of 5GB (which is the maximum file upload size), but we’d suggest specifying a smaller chunk size such as 500MB:

$ rclone copy archive_data.tar.gz orbit:my-container --swift-chunk-size=500M

Rclone will write the chunks to a separate Orbit container named <original_container_name>_segments. If you’re using a storage-only API client to authenticate then rclone won’t be able to automatically create this container and you’ll have to create that manually and grant the API client access to it.

There are a few other options related to large file support in the rclone documentation that are worth considering.

Alternative chunker overlay remote

Rclone also supports copying large files by creating a chunker overlay remote which sits in front of your actual remote and handles splitting large files up itself. This doesn’t need a separate container, but it does mean the files can only be read properly by rclone - you can’t download the large files directly.

To set up the overlay remote, edit your rclone config file:

[orbit]
type = swift
tenant = <account_id>
user = <api_client_id>
key = <api_client_secret>
auth = https://orbit.brightbox.com/v3

[orbit-chunker]
type = chunker
remote = orbit:/
chunk_size = 500M
hash_type = md5

You can then copy the files via the new overlay remote:

$ rclone copy archive_data.tar.gz orbit-chunker:my-container