Last month saw the coordinated disclosure of the CVE-2026-99999 “PwnEscape” vulnerability which affected a lot of providers, particularly anyone running virtual servers - like us :) Since then, a public proof of concept has been published. It’s a serious vulnerability but wasn’t too difficult get fixed.
If you have things hosted elsewhere (holds back tears how could you!?) you may have experienced hurried overnight emergency maintenance that caused noticeable reboots of your servers.
We were able to live patch this vulnerability without any interruption to service and I thought I’d explain how we do this, and what we do when it’s not possible.
In this case, we were able to develop and apply a patch to the running kernel on our Linux hosts using the kpatch feature. kpatch has been around for over 10 years now and applying patches is a very smooth process - the main challenge is developing the patch in the first place.
In this case the patch required no major changes so was easy to develop and we had a fix fully tested and out in production very quickly.
Where we’re not able to develop a safe kpatch for a vulnerability, we can instead use live migrations to move all cloud servers off a host without interrupting service (we even sync the conntrack table so established TCP don’t get disconnected). We have two main types of storage at Brightbox, local and network.
For network-storage servers, the live migration just copies the running server state (RAM etc) over to the new host in the background, does several passes to copy over anything that changed during the previous copy and then pauses the server to do a final sync, which usually takes just milliseconds, and unpauses it on the new host.
For local-storage servers, we have to copy the local disks first in a similar manner as the RAM, then the rest of the server state, so again, it is only paused for a few milliseconds.
We can then upgrade and reboot hosts without having to reboot any servers. An amusing result of this is that we have servers with considerably longer uptime than any of our host equipment! (though we do recommend customers do their own upgrades of course, and we manage that for our managed customers).
In the rare situation where we cannot live migrate a server, our fallback of course has to involve service interruption. Anyone affected is notified in advance and the downtime is kept to a minimum.
You can sign up for Brightbox in just a couple of minutes and get a ÂŁ50 free credit.
And if you’re looking for a UK-based team to manage your servers, drop us a line.
