Use the Kubernetes Cluster CA to Sign Each Node’s Kubelet Serving Certificate (kubeadm Bare-Metal Ubuntu 24.04)

If you run Metrics Server (or anything that connects to the Kubelet HTTPS endpoint on 10250), you may see TLS errors like:

• x509: certificate signed by unknown authority

That usually means your nodes are still using self-signed Kubelet serving certificates, so other in-cluster components can’t verify them.

This post is based on the following post:

The clean fix is to make each node’s Kubelet serving certificate come from the Kubernetes cluster CA, using the native Kubelet CSR flow:

• Enable serverTLSBootstrap: true on each node
• Kubelet generates a CSR with signer kubernetes.io/kubelet-serving
• You approve it manually
• Kubelet downloads and starts using the signed serving cert

This post covers only that (manual approval). No auto-approver, no --kubelet-insecure-tls.

Starting in Kubernetes v1.35, kubelet’s configuration drop-in directory (--config-dir) is GA, so we’ll use a drop-in file to keep serverTLSBootstrap: true outside the kubeadm-managed kubelet config.

Lab Context

Cluster built with kubeadm on bare-metal Ubuntu 24.04:

• k8s-1.maksonlee.com – 192.168.0.99
• k8s-2.maksonlee.com – 192.168.0.100
• k8s-3.maksonlee.com – 192.168.0.101
• Kubernetes: v1.35.x
• containerd + Calico

What This Changes (and What It Does Not)

Changes

• The Kubelet serving cert used by HTTPS on 10250 becomes cluster-signed
• The issued cert includes correct SANs (DNS + IP), e.g.:
  • DNS:k8s-1.maksonlee.com
  • IP Address:192.168.0.99

Does NOT change

• kubeadm join behavior
• The kubelet client certificate (node → apiserver); this is a different flow

Will the Cluster Keep Working While Doing This?

Yes. Do it one node at a time.

While a node is waiting for its serving CSR to be approved, the cluster still runs, but anything that needs kubelet 10250 TLS (metrics/logs/exec in some paths) can be flaky for that node until the cert is issued.

Why Use Drop-In Config Instead of Editing /var/lib/kubelet/config.yaml?

In kubeadm clusters, kubelet already runs with --config=/var/lib/kubelet/config.yaml, but kubeadm manages that file and may regenerate it during upgrades / node phases.

With kubelet drop-ins (--config-dir), kubelet loads your base config, then merges drop-ins on top (sorted by filename), then applies CLI flags last. That gives you a clean way to keep small overrides (like serverTLSBootstrap: true) separate from the kubeadm-managed file.

1. Confirm kubelet uses /var/lib/kubelet/config.yaml and supports overrides

On k8s-1:

sudo systemctl cat kubelet | sed -n '/KUBELET_CONFIG_ARGS/,$p'

You should see something like:

Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
EnvironmentFile=-/etc/default/kubelet
ExecStart=/usr/bin/kubelet ... $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

Key point:

• On Ubuntu (DEB), kubelet sources /etc/default/kubelet for KUBELET_EXTRA_ARGS, and it’s last in the flag chain.

2. Enable kubelet drop-in configuration directory (one node at a time)

On the node you’re working on (start with k8s-1):

Open:

sudo vi /etc/default/kubelet

If it’s empty like:

KUBELET_EXTRA_ARGS=

Change it to:

KUBELET_EXTRA_ARGS="--config-dir=/etc/kubernetes/kubelet.conf.d"

If it already contains arguments, keep them and append --config-dir=..., for example:

KUBELET_EXTRA_ARGS="--node-ip=192.168.0.99 --config-dir=/etc/kubernetes/kubelet.conf.d"

Restart kubelet:

sudo systemctl restart kubelet

3. Enable serverTLSBootstrap: true via a drop-in file (one node at a time)

On k8s-1:

Create the directory:

sudo mkdir -p /etc/kubernetes/kubelet.conf.d

Create a drop-in file (must end with .conf and must include apiVersion + kind):

cat <<'EOF' | sudo tee /etc/kubernetes/kubelet.conf.d/10-tls-bootstrap.conf
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
serverTLSBootstrap: true
EOF

Restart kubelet:

sudo systemctl restart kubelet

Notes:

• File must end with .conf
• File must include apiVersion and kind
• Files are processed in alphanumeric order; use a numeric prefix like 10-...
• Avoid leaving editor junk files (.swp, ~, .bak) in that directory

4. Verify serverTLSBootstrap is enabled (effective kubelet config)

kubectl get --raw "/api/v1/nodes/k8s-1.maksonlee.com/proxy/configz" \
  | jq '.kubeletconfig.serverTLSBootstrap'

Expected:

true

5. Find the Pending kubelet-serving CSR

From your admin terminal (k8s-1 is fine):

kubectl get csr -o wide | grep -E 'NAME|kubernetes.io/kubelet-serving|k8s-1'

You want a CSR where:

• SIGNERNAME = kubernetes.io/kubelet-serving
• REQUESTOR = system:node:k8s-1.maksonlee.com
• CONDITION = Pending

6. Approve the CSR Manually

Approve the CSR you saw above:

kubectl certificate approve <csr-name>

Verify:

kubectl get csr <csr-name> -o wide

Expected:

Approved,Issued

7. Verify the Node Installed the Cluster-Signed Serving Cert

On k8s-1:

sudo ls -l /var/lib/kubelet/pki/kubelet-server-current.pem
sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-server-current.pem -noout \
  -issuer -subject -dates -ext subjectAltName

Expected:

• Issuer should be your cluster CA (commonly CN = kubernetes in kubeadm clusters)
• SAN includes your node DNS + IP
• Validity dates show notBefore and notAfter

Example (good):

• issuer=CN = kubernetes
• DNS:k8s-1.maksonlee.com
• IP Address:192.168.0.99
• notAfter=Dec 29 12:38:30 2026 GMT (example)

8. Repeat for k8s-2 and k8s-3

Repeat the same procedure for k8s-2 and k8s-3.