🎉 Announcing new lower pricing — up to 40% lower costs for Cloud Servers and Cloud SQL! Read more →

How autoscaling Kubernetes on Brightbox works

We’ve just launched our Kubernetes autoscaler, which creates and destroys new Kubernetes cluster nodes in response to demand and I thought I’d go into more detail about how it all works.

Perhaps surprisingly, Kubernetes doesn’t include mechanisms to create and destroy resources on the cloud within the core code. Instead it delegates that task to components running on the cluster (like our Cloud Controller Manager which provides node information and controls access to the Load Balancers).

The component that adjusts the size of a Kubernetes is the Cluster Autoscaler, and we have created a version that you can run to automatically adjust the size of the cluster so that all pods have a place to run and there are no unneeded nodes.

How Autoscaler works on Brightbox

The autoscaler looks for Server Groups named after the cluster-name option passed to the autoscaler (--cluster-name).

A group named with a suffix of the cluster-name (e.g. k8s-worker.k8s-test.cluster.local) is a candidate to be a scaling group. The autoscaler will then check the description to see if it is a pair of integers separated by a colon (e.g. 1:4). If it finds those numbers then they will become the minimum and maximum server size for that group, and autoscaler will attempt to scale the group between those sizes.

The type of server, the image used and the target zone will be dynamically determined from the existing members. If these differ, or there are no existing servers, autoscaler will log an error and will not scale that group.

A group named precisely the same as the cluster-name (e.g. k8s-test.cluster.local) is considered to be the default cluster group and all autoscaled servers created are placed within it as well as the scaling group.

The Brightbox Autoscaler only supports auto-discovery mode using this pattern. node-group-auto-discovery and nodes options are effectively ignored.

Cluster configuration

If you are using the Kubernetes Cluster Builder set the worker_max values to scale the worker group, and the storage_max values to scale the storage group.

The Cluster Builder will ensure the group name and description are updated with the correct values in the format that autoscaler can recognise.

The Gory Details

Cluster Autoscaler works by listening to the Kubernetes scheduler and picking up on any unschedulable pod events. If it spots any it runs a ghost scheduling process against a changed list of nodes and if that would schedule the pods it asks the cloud provider to create the nodes. Similarly if it detects any idle resources it calculates a proposed smaller set of nodes and asks the scheduler whether it can reschedule the pods. If it can the node is drained and deleted on the cloud provider.

The main challenge has been updating our Kubernetes Cluster manifests so they create workers that will bootstrap themselves and connect to the cluster. Once that was in place we implemented a cloud driver in the Cluster Autoscaler and everything worked nicely.

However this is a compiled-in driver and means that we have to maintain a fork of the autoscaler repository. Because autoscaler uses the Kubernetes scheduler code that means maintaining a version for each current version of Kubernetes.

Obviously we’d rather avoid having to do this work, which is why we’re excited to see the autoscaler merge a Cluster API driver into the code. The ClusterAPI project allows each cloudprovider to expose its resources as native objects on Kubernetes Clusters allowing you to deploy cloud resources as you would normal Kubernetes pods, deployments or services.

We’ll be keeping an eye on this and see how the Cluster API matures.

Try Brightbox for free

If you want to play with Kubernetes, you can sign up for Brightbox real quick and use your £50 free credit to give it a go.

Managed Kubernetes services

If instead you want us to run Kubernetes for you, or anything else for that matter, we offer hands-on support and managed services too. Drop us a line.

Get started with Brightbox Sign up takes just two minutes...