Renewing Kubernetes Cluster Certificates

Kubernetes is supposed to renew its own certificates before there is a problem ... this was implemented in 1.15.x (??) ... but it is possible for there to be a problem if the master node is rebooted on the exact one-year anniversary date of the cluster: after the certs have expired, but before the renewal process is triggered. What you get is a message in the system logs:

Client rotation is on, will bootstrap in background
failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory

Very informative (not). After poking around, I found this post in Stack Overflow that had the answer -- summarized here ...

So the solution was to (first a backup) -- all commands as root:

mkdir ~/k8s
cd /etc/kubernetes/pki/
mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} ~/k8s/
kubeadm init phase certs all --apiserver-advertise-address <IP>
cd /etc/kubernetes/
mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} ~/k8s/
kubeadm init phase kubeconfig all
reboot

Then copy the new certs to the default location so we can use kubectl:

cp -i /etc/kubernetes/admin.conf $HOME/.kube/config