Upgrade a 3-Node kubeadm HA Kubernetes Cluster to v1.35.0 (kube-vip + MetalLB + Traefik) on Bare-Metal Ubuntu 24.04

This post shows how to upgrade a 3-node kubeadm cluster from Kubernetes v1.34.x → v1.35.0 on Ubuntu Server 24.04, where:

• kube-vip provides a floating API VIP (k8s.maksonlee.com → 192.168.0.97:6443)
• MetalLB (L2) provides a LoadBalancer IP (192.168.0.98)
• Traefik is the Ingress controller (3 replicas)

This upgrade follows the official kubeadm upgrade flow (don’t skip minor versions).

This post is based on

Lab context

Nodes (all control-plane + worker):

• k8s-1.maksonlee.com → 192.168.0.99
• k8s-2.maksonlee.com → 192.168.0.100
• k8s-3.maksonlee.com → 192.168.0.101

Networking:

• API VIP: k8s.maksonlee.com → 192.168.0.97:6443 (kube-vip)
• MetalLB LB IP: 192.168.0.98 (Traefik Service)

OS / runtime:

• Ubuntu Server 24.04.x
• containerd
• cgroup v2

Target:

• Kubernetes v1.35.0

1. Check current versions and cluster health

• Kubernetes versions

Run on your admin node (for example k8s-1):

kubectl version
kubeadm version
kubelet --version
kubectl get nodes -o wide

• containerd version

containerd --version

• Basic cluster health

kubectl get nodes
kubectl get pods -A
kubectl -n kube-system get pods -o wide | egrep 'etcd|kube-apiserver|kube-controller-manager|kube-scheduler|coredns|kube-proxy|calico|kube-vip|metrics'

2. Confirm the upgrade package version

On each node:

sudo apt update
apt policy kubeadm kubelet kubectl

(Optional) capture the candidate package version:

K8S_PKG_VER="$(apt-cache policy kubeadm | awk '/Candidate:/ {print $2}')"
echo "$K8S_PKG_VER"

3. Back up etcd (snapshot)

Because this is a stacked-etcd kubeadm cluster, etcd is the source of truth for cluster state. Before upgrading, take a snapshot from one control-plane node.

• Create a backup directory (host)

sudo mkdir -p /var/lib/etcd/backup

• Snapshot using the etcd static pod tools

Set the etcd pod name for the node you’re on (example: k8s-1):

ETCD_POD="etcd-k8s-1.maksonlee.com"
SNAP="/var/lib/etcd/backup/etcd-$(date +%F-%H%M).db"

kubectl -n kube-system exec "$ETCD_POD" -- etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  snapshot save "$SNAP"

• Verify snapshot looks valid

kubectl -n kube-system exec "$ETCD_POD" -- etcdutl --write-out=table snapshot status "$SNAP"

• Verify the file exists on the host

sudo ls -lh /var/lib/etcd/backup

4. Upgrade order

Upgrade one node at a time:

• k8s-1 (first control-plane)
• k8s-2
• k8s-3

This avoids losing etcd quorum and keeps the API available.

5. Upgrade the first control-plane node (k8s-1)

• Drain the node

From your admin machine:

kubectl drain k8s-1.maksonlee.com --ignore-daemonsets --delete-emptydir-data

• Upgrade kubeadm package (on k8s-1)

sudo apt-mark unhold kubeadm || true
sudo apt update
sudo apt install -y kubeadm="$K8S_PKG_VER"
sudo apt-mark hold kubeadm
kubeadm version

• Run the kubeadm control-plane upgrade (on k8s-1)

Use the official workflow: plan, then apply.

sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.35.0

• Upgrade kubelet + kubectl (on k8s-1)

sudo apt-mark unhold kubelet kubectl || true
sudo apt install -y kubelet="$K8S_PKG_VER" kubectl="$K8S_PKG_VER"
sudo apt-mark hold kubelet kubectl

sudo systemctl daemon-reload
sudo systemctl restart kubelet

• Uncordon and verify

Back on your admin machine:

kubectl uncordon k8s-1.maksonlee.com
kubectl get nodes -o wide
kubectl get pods -A

6. Upgrade the other control-plane nodes (k8s-2, k8s-3)

Repeat this section for k8s-2, then k8s-3.

• Drain the node

kubectl drain k8s-2.maksonlee.com --ignore-daemonsets --delete-emptydir-data

• Upgrade kubeadm (on that node)

sudo apt-mark unhold kubeadm || true
sudo apt update
sudo apt install -y kubeadm="$K8S_PKG_VER"
sudo apt-mark hold kubeadm
kubeadm version

• Run the node upgrade (on that node)

sudo kubeadm upgrade node

• Upgrade kubelet + kubectl (on that node)

sudo apt-mark unhold kubelet kubectl || true
sudo apt install -y kubelet="$K8S_PKG_VER" kubectl="$K8S_PKG_VER"
sudo apt-mark hold kubelet kubectl

sudo systemctl daemon-reload
sudo systemctl restart kubelet

• Uncordon and verify

kubectl uncordon k8s-2.maksonlee.com
kubectl get nodes -o wide

Do the same for k8s-3.

7. Post-upgrade verification checklist

• Cluster versions

kubectl version
kubectl get nodes -o wide
kubectl --server=https://k8s.maksonlee.com:6443 get nodes

• Core system pods

kubectl -n kube-system get pods -o wide | egrep \
'etcd|kube-apiserver|kube-controller-manager|kube-scheduler|coredns|kube-proxy|calico|kube-vip|metrics'

• MetalLB + Traefik

kubectl -n metallb-system get pods -o wide
kubectl -n traefik get pods -o wide
kubectl -n traefik get svc

8. Fix Traefik “2 pods on one node” after drain/uncordon

After draining nodes during upgrades, Traefik pods can end up unevenly placed (for example, 2 pods on one node). Kubernetes doesn’t “rebalance” existing pods when a node comes back.

The simplest fix (and what you already observed working): recreate Traefik pods via Helm:

helm upgrade traefik traefik/traefik -n traefik -f traefik-values.yaml
kubectl -n traefik get pods -o wide

(Alternative: kubectl -n traefik rollout restart deploy/traefik)

Done

At this point you should have:

• All nodes on v1.35.0
• kube-vip API VIP still working: k8s.maksonlee.com:6443
• MetalLB still advertising 192.168.0.98
• Traefik running as expected (and re-spread after upgrades)