OpenBao Deployment Guideline
This page is a non-binding guideline intended to help you get started with your own Key Management System for Volume Encryption. It is provided as-is, without warranty of any kind.
::: Deploying, operating, securing, upgrading, and backing up the KMS — including all key material it holds — is entirely your responsibility. T-Systems / Open Sovereign Cloud is neither responsible nor accountable for KMS deployments based on this guideline, and your KMS is not covered by OSC support or service level agreements.
OpenBao is the recommended first choice, but not the only one: volume encryption equally supports any KMIP-compatible KMS or HSM — the choice of key management stays yours.
This guideline deploys OpenBao as a 3-node High-Availability cluster spread across 3 availability zones inside your shoot cluster. It keeps serving through the loss of any one pod or one availability zone, planned or unplanned.
To tolerate two simultaneous failures, use 5 replicas (quorum = 3). The replica count must stay odd (3, 5, 7) for a stable quorum.
What this protects
- The key encryption key (KEK) lives only inside OpenBao
(Transit key with
exportable=false) — it never reaches a worker node or the Kubernetes API. - A dump of the cluster's Secrets yields only wrapped (encrypted) volume keys — useless without the KEK.
- The driver's token can only wrap and unwrap with the one key — it cannot read the key, touch other keys, or administer OpenBao.
- A holder of both the token and the wrapped-key Secrets could unwrap them until the token is revoked or expires. Treat the token Secret as sensitive, scope it least-privilege, and rotate it.
Prerequisites
- A shoot cluster with worker nodes in at least 3 availability zones
(nodes labelled
topology.kubernetes.io/zone). kubectland Helm ≥ 3.x.- A private CA to issue the OpenBao server certificate — TLS is mandatory for production (step 8). Public or ACME certificate authorities cannot issue certificates for ClusterIPs or cluster-internal DNS names, so use the cert-manager extension with a self-signed CA (shown in step 8) or your own PKI.
:::note Do not store OpenBao on encrypted volumes
OpenBao's own data volumes must use an unencrypted StorageClass
(for example default).
Using the encrypted StorageClass would create a circular dependency:
the volumes could never be attached while OpenBao is down.
All bao commands below run inside a server pod.
Define a helper once (it targets openbao-0; standby nodes forward to the leader):
After enabling TLS (step 8), the pods already point BAO_ADDR at https://…,
but the CLI additionally needs the CA —
change the helper's environment to
BAO_CACERT="/openbao/userconfig/openbao-server-tls/ca.crt"
(and drop the BAO_ADDR override).
1. Production values
Save as openbao-ha-values.yaml.
The comments mark the two production knobs (TLS, auto-unseal)
you wire in steps 3 and 8.
topologySpreadConstraints only helps if each topology.kubernetes.io/zone
value is a distinct physical availability zone.
An additional worker pool (for example a GPU pool) can carry its own
synthetic zone label while living in an existing zone —
the scheduler would treat it as an extra zone
and could place two Raft voters in one real zone.
The nodeSelector above pins OpenBao to one worker pool
whose zones are the real availability zones.
Verify after the install with kubectl -n openbao get pods -o wide.
2. Install
All three pods come up Running but sealed and not Ready until initialized (step 3) — that is expected. Confirm the spread is one pod per zone on three different nodes.
3. Initialize and unseal
Initialize once, on the first pod:
Unseal every pod — each node needs the threshold of keys, and the Raft followers join here:
Auto-unseal (strongly recommended)
With manual unseal, a restarted pod stays sealed until a human unseals it —
the cluster survives a failure but does not self-heal.
For hands-free recovery, configure a seal stanza (step 1)
before the first bao operator init:
seal "pkcs11"— an HSM or PKCS#11 token.seal "transit"— a separate, already-running OpenBao/Vault instance (do not point it at this cluster — that is circular).
Switching an already-initialized cluster to auto-unseal is a seal migration with brief downtime — follow the OpenBao documentation exactly.
4. Verify the Raft cluster
5. Enable Transit and create the key encryption key
Enable automatic key rotation (Transit keeps old key versions, so previously wrapped volume keys still decrypt):
6. Least-privilege policy
Grant only wrap and unwrap on the one key — nothing else:
7. Create the driver token
Mint a least-privilege, periodic token bound to the policy (the root token is used for setup only) and store it in the Secret referenced by your encrypted StorageClass:
Keep the token alive
A periodic token stays valid indefinitely, but only if it is renewed within each period. Renewal happens in OpenBao — the token value and the Kubernetes Secret stay unchanged, and no volume Secrets are involved. A minimal renewer:
If the token does expire:
no data is lost and running Pods keep working,
but attaching, rescheduling, and resizing encrypted volumes fail.
Mint a new token and update the kms-token Secret to recover.
Revoke the root token
After setup, revoke the root token and use a scoped operator token for day-2 operations (step 10):
After the root token is revoked,
admin operations (like changing policies) need a new root token.
OpenBao disables the classic unauthenticated generate-root endpoints
by default for security reasons,
and the bao operator generate-root command of OpenBao 2.5
still targets them — it fails with "unsupported operation".
To recover root access with your unseal keys,
temporarily set disable_unauthed_generate_root_endpoints = false
in the server configuration (restart required),
run bao operator generate-root,
and revert the setting afterwards.
Keep the ops token renewed so this stays a rare event.
8. Network, endpoint and TLS
- The storage driver cannot resolve cluster-internal DNS —
use a Service ClusterIP as the
encryptionKMSEndpoint, never a DNS name. - Use the
openbao-activeService, not the plainopenbaoone: it always points at the current Raft leader and never routes to a sealed node.
TLS — mandatory for production
Every unwrap request returns the plaintext key of a volume over this connection — running the KMS on plain HTTP is acceptable for a first functional bring-up, but not an option for production. Enable TLS before the first encrypted volume holds real data.
The server certificate must cover the openbao-active ClusterIP (IP SAN),
the pod DNS names openbao-{0,1,2}.openbao-internal (Raft join),
and 127.0.0.1 (in-pod CLI).
No public or ACME certificate authority can issue such a certificate,
so create a private CA.
With the cert-manager extension,
bootstrap a self-signed CA (long-lived, here 10 years)
and issue the server certificate from it (here 5 years):
Look up the ClusterIP the certificate must contain, then issue the server certificate:
Enable TLS in the chart
In openbao-ha-values.yaml:
set global.tlsDisable: false,
mount the certificate Secret,
and update the listener and the Raft retry_join stanzas:
Apply with helm upgrade, then restart the pods one at a time
(see Upgrades).
The endpoint becomes https://<openbao-active-clusterip>:8200 —
update the StorageClass accordingly
and hand the CA to the driver via the kms-token Secret:
The token-renewal CronJob (step 7) then also needs
BAO_ADDR=https://… and the CA:
mount the ca.crt key of the kms-token Secret
and point the BAO_CACERT environment variable at it.
cert-manager renews the certificate Secret automatically before expiry,
but OpenBao only reads the certificate files at startup:
after a renewal, perform a controlled rolling restart —
one pod at a time; with manual unseal, unseal each restarted pod again,
with auto-unseal it is hands-free.
With a 5-year server certificate this is a rare, plannable event.
Shortening the lifetime (for example to 1 year) is a good hardening step
as long as you operate that controlled restart process —
pair short lifetimes with auto-unseal.
One more trigger to plan for:
rotating the CA itself requires updating the ca.crt
in the kms-token Secret.
And a hard rule: never delete and recreate the openbao-active
Service — its ClusterIP is baked into the server certificate
and into every encrypted volume at creation time,
and existing volumes cannot follow an endpoint change.
9. Wire it to the StorageClass
Use the openbao-active ClusterIP as the endpoint and pvc-kek as the key —
see Volume Encryption
for the full StorageClass example.
The endpoint protocol must match the listener:
http:// while tlsDisable=true, https:// once TLS is enabled.
10. Operations
Disruption test
Prove the setup tolerates the loss of one pod — kill the leader, the hardest case:
With manual unseal, unseal the restarted pod to restore full redundancy; with auto-unseal it self-heals.
Backups
Losing the OpenBao storage (the KEK) means losing every encrypted volume. Snapshot on a schedule, store the snapshots off-cluster, and test the restore procedure.
Audit
The audit device is configured declaratively in the server configuration
(step 1) and writes to the persistent auditStorage volume —
OpenBao 2.5+ does not allow enabling audit devices via the API.
Every wrap and unwrap call is logged on the active node
at /openbao/audit/audit.log. Verify and ship the file off-cluster:
Upgrades
With updateStrategyType: OnDelete (manual unseal), roll one pod at a time:
delete a pod, wait for it to run, unseal it, confirm it is Ready, then the next.
With auto-unseal, switch to RollingUpdate and let the chart roll them.
Production checklist
Availability
- 3 Raft replicas — one per availability zone, no two on the same node
PodDisruptionBudgetwithmaxUnavailable: 1- Auto-unseal configured before the first initialization
- Disruption test passed — one pod killed, volumes still attach
Security
- TLS enabled with a private CA (~10 years) and a server certificate (~5 years, or shorter with a controlled restart process)
- CA handed to the driver via the
ca.crtkey of thekms-tokenSecret - Key encryption key not exportable, automatic rotation enabled
- Driver token least-privilege (wrap and unwrap only) and periodic, with a renewer or a rotation process
- Root token revoked after setup
Operations
- StorageClass endpoint is the
openbao-activeService ClusterIP, protocol matching the listener (https://with TLS) - Audit device configured declaratively in the server config (step 1), log on a persistent volume, shipped off-cluster
- Raft snapshots scheduled, stored off-cluster, restore tested