Reviving 2 years old Kubernetes cluster

README.md

Scenario

1. etcdadm managed etcd cluster
2. kubeadm managed kubernetes cluster
3. all etcd certificates have been expired (self-signed CA is okay)
4. all kubernetes certificates have been expired

Recovery

Note: Do not forget to make backup before changing any files.

1. stop all relevant systemd units on all nodes:

   # systemctl stop etcd.service
   # systemctl stop kubelet.service
   

2. renew etcd certificates on all nodes:

   # for crt in /etc/etcd/pki/*.crt ; do echo "$crt:" ; openssl x509 -noout -dates -in $crt ; echo ; done
   # find /etc/etcd/pki -type f ! -iname 'ca.crt' ! -iname 'ca.key' -exec echo {} \;
   # find /etc/etcd/pki -type f ! -iname 'ca.crt' ! -iname 'ca.key' -delete
   # etcdadm join phase certificates https://lan.ip.goes.here:2379
   # systemctl start etcd.service
   # systemctl status etcd.service
   # source /etc/etcd/etcdctl.env
   # etcdctl member list
   

3. renew kubernetes certificates (control plane nodes):

   # kubeadm certs check-expiration --config /home/ubuntu/projects/kubeadmcfg-external.yaml
   # rm -f /var/lib/kubelet/pki/kubelet.crt
   # rm -f /var/lib/kubelet/pki/kubelet.key
   # rm -f /var/lib/kubelet/pki/kubelet-client*
   # rm -f /etc/kubernetes/admin.conf
   # rm -f /etc/kubernetes/controller-manager.conf
   # rm -f /etc/kubernetes/kubelet.conf
   # rm -f /etc/kubernetes/scheduler.conf
   # kubeadm init phase kubelet-finalize all --config /home/ubuntu/projects/kubeadmcfg-external.yaml
   # kubeadm certs renew all --config /home/ubuntu/projects/kubeadmcfg-external.yaml
   # kubeadm init phase kubeconfig all --config /home/ubuntu/projects/kubeadmcfg-external.yaml
   # kubeadm kubeconfig user --org system:nodes --client-name system:node:$(hostname) --config /home/ubuntu/projects/kubeadmcfg-external.yaml > /etc/kubernetes/kubelet.conf
   # systemctl start kubelet.service
   # systemctl status kubelet.service
   # kubeadm token create --print-join-command
   

4. renew kubernetes certificates (worker nodes):

   # kubeadm reset -f --cri-socket unix:///var/run/cri-dockerd.sock
   # kubeadm join api.server.hostname:6443 --token join.token.goes.here --discovery-token-ca-cert-hash sha256:certificate.hash.goes.here --cri-socket unix:///var/run/cri-dockerd.sock --ignore-preflight-errors=FileAvailable--etc-kubernetes-pki-ca.crt
   # systemctl start kubelet.service
   # systemctl status kubelet.service
   

5. check status:

   $ kubectl get nodes
   NAME                 STATUS   ROLES           AGE     VERSION
   inst-biuce-vmp-prv   Ready    <none>          2y88d   v1.24.10
   inst-elb5m-vmp-pub   Ready    control-plane   2y95d   v1.24.10
   inst-tjvsi-vmp-pub   Ready    control-plane   2y95d   v1.24.10
   

References:

• kubernetes/kubernetes#110035 (comment)
• kubernetes/kubeadm#581 (comment)
• kubernetes/kubeadm#1447
• https://blog.scottlowe.org/2019/07/30/adding-a-name-to-kubernetes-api-server-certificate/

# cd /etc/kubernetes/pki/
# rm -f apiserver.*
# rm -f apiserver-kubelet-client.*
# rm -f front-proxy-client.*
# kubeadm init phase certs all --config /home/ubuntu/projects/kubeadmcfg-external.yaml