Sync HashiCorp Vault Secrets into Kubernetes using External Secrets Operator (ESO)

By /

This post shows how to sync secrets from HashiCorp Vault OSS into a kubeadm Kubernetes cluster using External Secrets Operator (ESO), and consume them as a normal Kubernetes Secret using:

envFrom:
  - secretRef:
      name: backstage-env

We’ll use Backstage only as a real-world example (backstage-env), but the workflow applies to any app.

This post is based on:

Build a 3-Node HA Kubernetes Cluster with kube-vip, MetalLB, and Traefik on Bare-Metal Ubuntu 24.04
How to Install HashiCorp Vault OSS on AWS EC2 (Ubuntu 24.04)

Lab context

  • Vault OSS (Ubuntu 24.04)
    • Vault listens on 127.0.0.1:8200
    • NGINX + Certbot exposes HTTPS: https://vault.maksonlee.com
    • Secrets engine: KV v2 mounted at secret/
    • Auth method: auth/kubernetes
  • Kubernetes: kubeadm bare-metal cluster (Ubuntu 24.04)
    • API endpoint: https://k8s.maksonlee.com:6443 (kube-vip VIP)

What you’ll build

  • Install External Secrets Operator (ESO)
  • Configure Vault Kubernetes Auth (one-time)
  • Store an env-style secret in Vault at secret/<app>/<env>
  • Create a namespaced SecretStore
  • Create an ExternalSecret to generate/update a Kubernetes Secret
  • Consume the Secret using envFrom
  • Verify everything end-to-end

Architecture (ESO ↔ Vault ↔ Kubernetes API)

Prerequisites

  • kubectl access to the cluster
  • helm installed
  • Vault admin access on the Vault host
  • Network/DNS:
    • Kubernetes → https://vault.maksonlee.com
    • Vault → https://k8s.maksonlee.com:6443
  1. Sanity checks
  • Kubernetes → Vault API

From any Kubernetes node:

curl -s https://vault.maksonlee.com/v1/sys/health | head

You should get JSON.

  • Vault → Kubernetes API

From Vault :

curl -k https://k8s.maksonlee.com:6443/version

-k is fine for a quick test. Vault itself will validate TLS using the CA cert you configure later.

  1. Install External Secrets Operator (ESO)
helm repo add external-secrets https://charts.external-secrets.io
helm repo update

helm install external-secrets external-secrets/external-secrets \
  -n external-secrets --create-namespace \
  --set installCRDs=true

Verify:

kubectl get pods -n external-secrets
kubectl api-resources | grep -i externalsecret
kubectl api-resources | grep -i secretstore
  1. Kubernetes: create TokenReviewer SA (for Vault)

Vault needs a ServiceAccount token that can call TokenReview.

  • Create ServiceAccount + ClusterRoleBinding
kubectl -n external-secrets create serviceaccount vault-reviewer

kubectl create clusterrolebinding vault-reviewer-auth-delegator \
  --clusterrole=system:auth-delegator \
  --serviceaccount=external-secrets:vault-reviewer
  • Create a stable service-account-token Secret

Create vault-reviewer-token.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: vault-reviewer-token
  namespace: external-secrets
  annotations:
    kubernetes.io/service-account.name: vault-reviewer
type: kubernetes.io/service-account-token

Apply:

kubectl apply -f vault-reviewer-token.yaml

Extract the reviewer JWT + cluster CA:

kubectl -n external-secrets get secret vault-reviewer-token -o jsonpath='{.data.token}' \
  | base64 -d > ~/vault-token-reviewer.jwt

kubectl -n external-secrets get configmap kube-root-ca.crt -o jsonpath='{.data.ca\.crt}' \
  > ~/k8s-ca.crt
  1. Vault: enable Kubernetes auth and bind to your cluster

On Vault:

vault login
vault auth enable kubernetes || true

From the machine where you extracted the files, copy them to the Vault host:

scp ~/vault-token-reviewer.jwt ~/k8s-ca.crt ubuntu@vault.maksonlee.com:/tmp/

Configure Vault Kubernetes auth:

vault write auth/kubernetes/config \
  token_reviewer_jwt="$(cat /tmp/vault-token-reviewer.jwt)" \
  kubernetes_host="https://k8s.maksonlee.com:6443" \
  kubernetes_ca_cert=@/tmp/k8s-ca.crt

Verify:

vault read auth/kubernetes/config
  1. Vault: store an env-style secret (KV v2)

Example env keys:

AUTH_SESSION_SECRET=REPLACE_ME
AUTH_OIDC_CLIENT_SECRET=REPLACE_ME
LDAP_BIND_PASSWORD=REPLACE_ME
GITHUB_TOKEN=REPLACE_ME
GERRIT_TOKEN=REPLACE_ME
JENKINS_API_TOKEN=REPLACE_ME
ARTIFACTORY_AUTH_HEADER=REPLACE_ME
POSTGRES_PASSWORD=REPLACE_ME
ARGOCD_AUTH_TOKEN=<RAW_JWT_TOKEN>

Best way: JSON file → vault kv put ... @file.json

Create /tmp/backstage-prod.json:

cat > /tmp/backstage-prod.json <<'EOF'
{
  "AUTH_SESSION_SECRET": "REPLACE_ME",
  "AUTH_OIDC_CLIENT_SECRET": "REPLACE_ME",
  "LDAP_BIND_PASSWORD": "REPLACE_ME",
  "GITHUB_TOKEN": "REPLACE_ME",
  "GERRIT_TOKEN": "REPLACE_ME",
  "JENKINS_API_TOKEN": "REPLACE_ME",
  "ARTIFACTORY_AUTH_HEADER": "REPLACE_ME",
  "POSTGRES_PASSWORD": "REPLACE_ME",
  "ARGOCD_AUTH_TOKEN": "<RAW_JWT_TOKEN>"
}
EOF

Write to Vault (example path secret/backstage/prod):

vault login
vault kv put secret/backstage/prod @/tmp/backstage-prod.json
vault kv get secret/backstage/prod
  1. Vault: policy + role (least privilege)

Policy (KV v2 paths)

Create eso-read-backstage.hcl on Vault:

cat > eso-read-backstage.hcl <<'EOF'
path "secret/data/backstage/prod" {
  capabilities = ["read"]
}

path "secret/metadata/backstage/*" {
  capabilities = ["list"]
}
EOF

vault policy write eso-read-backstage eso-read-backstage.hcl

Role bound to a Kubernetes ServiceAccount

We’ll bind a role to:

  • Namespace: backstage
  • ServiceAccount: eso-vault
  • Audience: vault
vault write auth/kubernetes/role/eso-backstage \
  bound_service_account_names="eso-vault" \
  bound_service_account_namespaces="backstage" \
  policies="eso-read-backstage" \
  audience="vault" \
  ttl="1h"
  1. Kubernetes: ServiceAccount
kubectl -n backstage create serviceaccount eso-vault --dry-run=client -o yaml | kubectl apply -f -
  1. Kubernetes: SecretStore (namespaced)

In your ESO CRD, audiences is under serviceAccountRef (not directly under kubernetes).

Confirm:

kubectl explain secretstore.spec.provider.vault.auth.kubernetes --recursive

Create secretstore-vault-backstage.yaml:

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: backstage
spec:
  provider:
    vault:
      server: "https://vault.maksonlee.com"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "eso-backstage"
          serviceAccountRef:
            name: "eso-vault"
            audiences:
              - vault

Apply:

kubectl apply -f secretstore-vault-backstage.yaml
kubectl -n backstage describe secretstore vault-backend
  1. Kubernetes: ExternalSecret → Secret/backstage-env

Create externalsecret-backstage-env.yaml:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: backstage-env
  namespace: backstage
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: backstage-env
    creationPolicy: Owner
    deletionPolicy: Retain
  dataFrom:
    - extract:
        key: backstage/prod

Apply:

kubectl apply -f externalsecret-backstage-env.yaml
  1. Workload: consume the Secret via envFrom

Example snippet:

envFrom:
  - secretRef:
      name: backstage-env

Restart the Deployment once so pods load env vars:

kubectl -n backstage rollout restart deploy <your-deployment>
kubectl -n backstage rollout status deploy <your-deployment>
  1. Verification (end-to-end)

Verify ESO is syncing and managing the Secret

kubectl -n backstage get externalsecret backstage-env -o wide
kubectl -n backstage describe externalsecret backstage-env | tail -n 40

You want:

  • STATUS SecretSynced
  • READY True

Strong proof: delete the Secret and watch ESO recreate it

Delete:

kubectl -n backstage delete secret backstage-env

Watch it come back:

kubectl -n backstage get secret backstage-env -w

Confirm keys only (no values)

kubectl -n backstage get secret backstage-env -o jsonpath='{.data}' \
| python3 -c 'import sys,json; print("\n".join(sorted(json.load(sys.stdin).keys())))'

Did this guide save you time?

Support this site

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top