Previously, we explained how to securely install the Kubernetes Metrics Server. However, the Kubernetes Controller Manager will only approve certificate signing requests for client certificates necessary to connect to the API server. Signing requests for server certificates requires manual approval, both initially and for each renewal.
Which tends to be rather inconvenient.
This default behaviour is meant to protect against a compromised server generating arbitrary certificates and mounting Man in the Middle attacks within the cluster. But the reasoning for this default behaviour is a bit outdated, so we can enable auto-signing now, as long as we’re careful.
There are third party operators available which will sign the server requests for you. The one we are using is Kubelet Rubber Stamp. This approves a kubelet certificate signing request (CSR) as long as the requestor id passes a Subject Access Review which ensures the node has the correct authorization to request certificates.
Rubber Stamp will only accept certificate requests where the Common Name matches
the authorized requestor id, the Common Name has a system:node: prefix and the
Organisation is system:nodes. The requestor id (in our case, the hostname)
comes from kubelet’s client certificate, which is already authorised and signed
so we can trust it. And those common name prefixes mean that even if an attacker
can somehow obtain a server certificate with an arbitrary name, they are clearly
only kubelet certificates.
This all prevents a compromised server generating certificates for arbitrary services within the cluster, so the man is kept out of the middle.
Let’s enable the auto-signer.
Clone the Kubelet Rubber Stamp repository and apply the deploy directory
$ git clone git@github.com:kontena/kubelet-rubber-stamp.git /tmp/stamp
$ kubectl apply -f /tmp/stamp/deployOnce installed, the operator will approve all outstanding server CSRs and any new ones that kubelet creates during certificate renewal.
You can view the logs to check the certificates are being signed as expected
$ kubectl -n kube-system logs deployment/kubelet-rubber-stamp
I0916 14:41:02.068037 1 main.go:19] Go Version: go1.11.13
I0916 14:41:02.068320 1 main.go:20] Go OS/Arch: linux/amd64
I0916 14:41:02.068503 1 main.go:21] operator-sdk Version: v0.4.0
I0916 14:41:02.478254 1 main.go:49] Registering Components.
I0916 14:41:02.483060 1 main.go:61] Starting the Cmd.
I0916 14:41:02.784772 1 certificatesigningrequest_controller.go:84]
Reconciling CertificateSigningRequest /csr-hnb48
I0916 14:41:02.792646 1 certificatesigningrequest_controller.go:131]
approving csr csr-hnb48 with SANs: [srv-krwu8], IP Addresses:[10.241.213.90]
I0916 14:41:02.805376 1 certificatesigningrequest_controller.go:84]
Reconciling CertificateSigningRequest /csr-554ml
I0916 14:41:02.810093 1 certificatesigningrequest_controller.go:131]
approving csr csr-554ml with SANs: [srv-krwu8], IP Addresses:[10.241.213.90]
I0916 14:41:02.819700 1 certificatesigningrequest_controller.go:84]
Reconciling CertificateSigningRequest /csr-6qm2f
I0916 14:41:02.819892 1 certificatesigningrequest_controller.go:101]
CSR already has a certificate, ignoringIf you want to play with Kubernetes, you can sign up for Brightbox Cloud in just a couple of minutes and use your £50 free credit to give it a go.
If instead you want us to run Kubernetes for you, or anything else for that matter, we offer hands-on support and managed services too. Drop us a line.