Use External Ceph RBD as Dynamic Storage in Kubernetes

In previous posts,

we:

• Built a 3-node HA Kubernetes cluster with kube-vip, MetalLB, and Traefik on bare-metal Ubuntu 24.04.
• Built an All-in-One Ceph S3 Test Environment on a Single Node on ceph.maksonlee.com (Ceph Squid v19, RGW, Dashboard, HAProxy, Let’s Encrypt).

This post is the next step: use that same Ceph node as dynamic block storage for Kubernetes via Ceph CSI RBD.

We will:

• Create a dedicated RBD pool for Kubernetes.
• Create a CephX client user restricted to that pool.
• Configure Ceph CSI with ConfigMaps + Secret.
• Deploy the Ceph CSI RBD driver (v3.15.1).
• Create a StorageClass that dynamically provisions RBD-backed PersistentVolumes.
• Test with a PVC + Pod and verify the RBD image from the Ceph side.

No Rook. No in-tree RBD (deprecated). Just kubeadm + ceph-csi + an existing single-node Ceph cluster.

Lab Overview

Kubernetes Cluster (Bare-Metal HA)

From Build a 3-Node HA Kubernetes Cluster with kube-vip, MetalLB, and Traefik on Bare-Metal Ubuntu 24.04:

• OS: Ubuntu Server 24.04 on all three nodes
• Kubernetes: v1.34 via kubeadm (pkgs.k8s.io)
• Container runtime: containerd.io (Docker apt repo), SystemdCgroup = true
• Pod CIDR: 10.244.0.0/16 (Calico)
• Service CIDR: 10.96.0.0/12 (kubeadm default)

Nodes (control-plane + worker, stacked etcd):

• k8s-1.maksonlee.com – 192.168.0.99
• k8s-2.maksonlee.com – 192.168.0.100
• k8s-3.maksonlee.com – 192.168.0.101

VIPs and ingress:

• API VIP (kube-vip):
  • k8s.maksonlee.com → 192.168.0.97:6443
• MetalLB (L2 mode):
  • IP pool: 192.168.0.98-192.168.0.98 (Traefik LoadBalancer Service)
• Ingress: Traefik (3 replicas, one per node) exposed at 192.168.0.98

All three nodes are schedulable for workloads (control-plane taints removed).

Ceph Cluster (Single-Node S3 Test Environment)

From All-in-One Ceph S3 Test Environment on a Single Node:

Key properties:

• OS: Ubuntu 24.04
• Ceph: Squid v19 via cephadm
• Single OSD node, pool size = 1 (lab-only)
• RGW and Dashboard fronted by HAProxy with Let’s Encrypt TLS (ceph.maksonlee.com)

That environment focused on S3 / RGW.

In this post, we:

• Reuse the same node ceph.maksonlee.com
• Add an RBD pool for Kubernetes
• Integrate it as dynamic storage for the 3-node kubeadm cluster

Why Ceph RBD for Kubernetes?

Ceph RBD + CSI gives you:

• Durable block volumes (within your Ceph replication settings; in this lab, size=1).
• Dynamic provisioning: Kubernetes creates/deletes RBD images as PVCs come and go.
• Decoupled storage: data lives in Ceph, not on any single Kubernetes node.
• Filesystem or raw block (volumeMode: Filesystem or Block).

The flow:

• Create a Ceph pool for Kubernetes.
• Create a CephX user with access only to that pool.
• Configure Ceph CSI (ConfigMaps + Secret).
• Deploy Ceph CSI RBD driver.
• Create a StorageClass.
• Test with PVC + Pod and verify from the Ceph side.

1. Create the Kubernetes RBD Pool on Ceph

Create an RBD pool named kubernetes:

sudo cephadm shell -- \
  ceph osd pool create kubernetes 8

Initialize it for RBD:

sudo cephadm shell -- \
  rbd pool init kubernetes

Verify:

sudo cephadm shell -- ceph osd pool ls detail | grep kubernetes

In this single-node lab (pool size = 1), 8 PGs is fine to start. For a real multi-node Ceph, calculate PGs properly.

2. Create CephX User for Kubernetes / Ceph CSI

Create client.kubernetes with profile rbd caps restricted to the kubernetes pool.

On the Ceph node:

sudo cephadm shell -- \
  ceph auth get-or-create client.kubernetes \
    mon 'profile rbd' \
    osd 'profile rbd pool=kubernetes' \
    mgr 'profile rbd pool=kubernetes'

The command prints just the user and key:

[client.kubernetes]
        key = AQDexampleKeyxxxxxxxxxxxxxxxxxxxxxxxxxxxx==

Write down:

• User ID: kubernetes
• Key: the AQ...== string from your cluster

To see the full record (including caps), run:

sudo cephadm shell -- \
  ceph auth get client.kubernetes

Example full output:

[client.kubernetes]
        key = AQDexampleKeyxxxxxxxxxxxxxxxxxxxxxxxxxxxx==
        caps mgr = "profile rbd pool=kubernetes"
        caps mon = "profile rbd"
        caps osd = "profile rbd pool=kubernetes"

The important part for Kubernetes integration is that client.kubernetes has profile rbd on the kubernetes pool.

3. Get Ceph FSID and MON v1 Address

Ceph CSI needs:

• clusterID → the Ceph FSID
• monitors → list of MON endpoints using v1 protocol (:6789)

On the Ceph node:

sudo cephadm shell -- ceph mon dump

In this lab, the output looks like:

epoch 1
fsid b3c4f78b-8e4a-11f0-943a-525400afa471
last_changed 2025-09-10T13:34:15.648877+0000
created 2025-09-10T13:34:15.648877+0000
min_mon_release 19 (squid)
election_strategy: 1
0: [v2:192.168.0.81:3300/0,v1:192.168.0.81:6789/0] mon.ceph
dumped monmap epoch 1

From this:

• clusterID (FSID) → b3c4f78b-8e4a-11f0-943a-525400afa471
• MON v1 address → 192.168.0.81:6789

We’ll plug these into the ceph-csi-config and StorageClass.

4. Install ceph-common and Load the rbd Kernel Module

Each Kubernetes node needs:

• ceph-common client tools
• The rbd kernel module for kernel RBD (krbd)

On all three Kubernetes nodes:

# Install Ceph client tools
sudo apt update
sudo apt install -y ceph-common

# Load RBD kernel module now
sudo modprobe rbd

# Make module persistent across reboots
echo rbd | sudo tee /etc/modules-load.d/rbd.conf

Verify:

lsmod | grep rbd || echo "rbd module not loaded"

The Ceph CSI RBD driver uses the kernel RBD mounter by default.
The userspace rbd-nbd mounter is only used if you explicitly set mounter: rbd-nbd in the StorageClass and install nbd + rbd-nbd on the nodes.

5. Configure Ceph CSI (ConfigMaps + Secret)

Configure the Kubernetes-side objects Ceph CSI needs:

• ceph-csi-config – Ceph FSID + MONs
• ceph-csi-encryption-kms-config – KMS config (empty for now)
• ceph-config – minimal ceph.conf + keyring stub
• csi-rbd-secret – CephX credentials for client.kubernetes

All commands below run on k8s-1.

For simplicity, everything lives in the default namespace in this lab.

• ceph-csi-config (FSID + MONs)

Create csi-config-map.yaml:

cat <<EOF > csi-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: ceph-csi-config
data:
  config.json: |-
    [
      {
        "clusterID": "b3c4f78b-8e4a-11f0-943a-525400afa471",
        "monitors": [
          "192.168.0.81:6789"
        ]
      }
    ]
EOF

Apply:

kubectl apply -f csi-config-map.yaml
kubectl get configmap ceph-csi-config

• KMS Config (Empty)

Create csi-kms-config-map.yaml:

cat <<EOF > csi-kms-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: ceph-csi-encryption-kms-config
data:
  config.json: |-
    {}
EOF

kubectl apply -f csi-kms-config-map.yaml
kubectl get configmap ceph-csi-encryption-kms-config

• ceph-config ConfigMap (ceph.conf + keyring stub)

Create ceph-config-map.yaml:

cat <<EOF > ceph-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: ceph-config
data:
  ceph.conf: |
    [global]
    auth_cluster_required = cephx
    auth_service_required = cephx
    auth_client_required  = cephx
  keyring: |
EOF

Apply:

kubectl apply -f ceph-config-map.yaml
kubectl get configmap ceph-config

• CephX Secret for CSI (client.kubernetes)

On the Ceph node, get only the key value:

sudo cephadm shell -- ceph auth get-key client.kubernetes

That prints something like:

AQDexampleKeyxxxxxxxxxxxxxxxxxxxxxxxxxxxx==

Back on k8s-1, create csi-rbd-secret.yaml:

cat <<EOF > csi-rbd-secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: default
stringData:
  userID: kubernetes
  userKey: <paste output of: ceph auth get-key client.kubernetes>
EOF

Apply:

kubectl apply -f csi-rbd-secret.yaml
kubectl get secret csi-rbd-secret

Treat this key like any other secret: store it safely and avoid committing it to public repos.

6. Deploy the Ceph CSI RBD Driver (v3.15.1)

Deploy Ceph CSI RBD from the v3.15.1 tag so the manifests and container image all match.

At the time of writing, Ceph CSI v3.15.x is tested with Kubernetes v1.31–v1.33.
This lab runs v1.34, which is slightly newer. For a homelab this is fine; for strict production, stay within the tested compatibility matrix.

• Clone the Ceph-CSI Repo at v3.15.1

On k8s-1:

cd ~
git clone --branch v3.15.1 https://github.com/ceph/ceph-csi.git
cd ceph-csi

RBD manifests are under deploy/rbd/kubernetes/.

• Apply RBAC for RBD

Apply RBAC for the RBD CSI controller and node plugin:

kubectl apply -f deploy/rbd/kubernetes/csi-provisioner-rbac.yaml
kubectl apply -f deploy/rbd/kubernetes/csi-nodeplugin-rbac.yaml

• Deploy the RBD CSI Provisioner and Node Plugin

Apply the main RBD CSI controller and node plugin manifests:

kubectl apply -f deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml
kubectl apply -f deploy/rbd/kubernetes/csi-rbdplugin.yaml

In the v3.15.1 tag, these YAMLs already use the matching image tag, for example:

image: quay.io/cephcsi/cephcsi:v3.15.1

Check:

kubectl get pods -A | grep csi-rbd

You should see:

• One or more csi-rbdplugin-provisioner pods (controller)
• A csi-rbdplugin DaemonSet pod on each node

Wait until everything is Running and Ready.

7. Create a StorageClass for Ceph RBD

Create a StorageClass that ties everything together:

• Provisioner: rbd.csi.ceph.com
• clusterID: b3c4f78b-8e4a-11f0-943a-525400afa471
• pool: kubernetes
• CSI secrets: csi-rbd-secret in default namespace

Create csi-rbd-sc.yaml:

cat <<EOF > csi-rbd-sc.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
  # Must match FSID from 'ceph mon dump'
  clusterID: b3c4f78b-8e4a-11f0-943a-525400afa471

  # Our Kubernetes pool
  pool: kubernetes

  # Enable layering for snapshots/clones
  imageFeatures: layering

  # CSI secrets
  csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
  csi.storage.k8s.io/provisioner-secret-namespace: default
  csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: default
  csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
  csi.storage.k8s.io/node-stage-secret-namespace: default

reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - discard
EOF

kubectl apply -f csi-rbd-sc.yaml
kubectl get storageclass

You should see csi-rbd-sc in the list.

To make it the default StorageClass:

kubectl patch storageclass csi-rbd-sc \
  -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

8. Test with PVC + Pod (Filesystem Volume)

Verify that:

• Creating a PVC with storageClassName: csi-rbd-sc creates an RBD image in the kubernetes pool.
• A Pod can mount it and write data.
• Deleting the PVC removes the RBD image (reclaimPolicy: Delete).

• Create a Filesystem PVC

Create rbd-pvc.yaml:

cat <<EOF > rbd-pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 10Gi
  storageClassName: csi-rbd-sc
EOF

kubectl apply -f rbd-pvc.yaml
kubectl get pvc rbd-pvc

Wait until the PVC status is Bound.

On the Ceph node:

sudo cephadm shell -- rbd ls kubernetes

You should see a new RBD image with a name like csi-vol-xxxx....

Inspect it:

sudo cephadm shell -- \
  rbd info kubernetes/<your-image-name>

• Mount It in a Test Pod

Create rbd-pod.yaml:

cat <<EOF > rbd-pod.yaml
---
apiVersion: v1
kind: Pod
metadata:
  name: csi-rbd-demo-pod
spec:
  containers:
    - name: web-server
      image: nginx
      volumeMounts:
        - name: mypvc
          mountPath: /usr/share/nginx/html
  volumes:
    - name: mypvc
      persistentVolumeClaim:
        claimName: rbd-pvc
        readOnly: false
EOF

Apply and check:

kubectl apply -f rbd-pod.yaml
kubectl get pod csi-rbd-demo-pod

When it’s Running:

kubectl exec -it csi-rbd-demo-pod -- sh

# Inside container:
df -h /usr/share/nginx/html
echo "hello from ceph rbd on k8s" > /usr/share/nginx/html/index.html
ls -l /usr/share/nginx/html
exit

If desired, you can expose this Pod via your existing Traefik + MetalLB setup so app1.maksonlee.com / app2.maksonlee.com serve pages from Ceph-backed storage.

• Clean Up and Confirm Reclaim

Delete the Pod and PVC:

kubectl delete pod csi-rbd-demo-pod
kubectl delete pvc rbd-pvc

Back on the Ceph node:

sudo cephadm shell -- rbd ls kubernetes

The RBD image used for that PVC should now be gone.