HashiCorp Vault in Production: Secrets Management for Kubernetes

Kubernetes Secrets are base64 encoded, not encrypted. Here's how to set up Vault properly — Kubernetes auth, agent injection, secret rotation without restarts, and the audit trail regulated environments require.

GK
Gaurav Kaushal Lead DevOps Engineer

Why Kubernetes secrets aren’t enough

Kubernetes Secrets are base64 encoded, not encrypted. Anyone with read access to the etcd database or sufficient RBAC permissions can read every secret in your cluster in plaintext. For many organisations this is an acceptable risk with proper RBAC. For anything handling financial data, PII, or operating in a regulated environment, it isn’t.

HashiCorp Vault solves the problem properly — secrets are encrypted at rest and in transit, access is audited, secrets can be rotated without application restarts, and you get a complete audit log of who accessed what and when. Here’s how to set it up and integrate it with Kubernetes in a production-viable way.

Vault architecture for Kubernetes

The integration works through the Vault Agent Injector — a mutating webhook that intercepts pod creation and injects a Vault Agent sidecar. The sidecar authenticates to Vault using the pod’s Kubernetes service account, fetches the required secrets, and writes them to a shared in-memory volume. Your application reads secrets from files, not environment variables.

# Install Vault via Helm
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

helm install vault hashicorp/vault   --namespace vault   --create-namespace   --set "server.ha.enabled=true"   --set "server.ha.replicas=3"   --set "injector.enabled=true"

# Initialize Vault (first time only)
kubectl exec -n vault vault-0 -- vault operator init   -key-shares=5   -key-threshold=3   -format=json > vault-init.json

Critical: Store the unseal keys and root token from vault-init.json somewhere secure immediately — AWS Secrets Manager, a password manager, encrypted storage. If you lose these, you lose access to Vault permanently. Delete the local file after storing safely.

Configuring Kubernetes authentication

# Enable Kubernetes auth method
vault auth enable kubernetes

# Configure it with the cluster's API server
vault write auth/kubernetes/config   kubernetes_host="https://$(kubectl get svc kubernetes -o jsonpath='{.spec.clusterIP}'):443"   kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt   token_reviewer_jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token

# Create a policy for your application
vault policy write api-service - 

**Worth knowing:** The audit log is append-only and cannot be disabled once Vault has processed a request with it enabled. This is intentional — it means even a compromised Vault administrator cannot retroactively hide access events.
GK
Gaurav Kaushal
Lead DevOps Engineer · AWS SA Professional · CKA in progress

8+ years managing large-scale infrastructure, CI/CD systems, and Kubernetes clusters. I write about what I've learned the hard way — production lessons, not docs rewrites.

Newsletter
More like this, in your inbox.
No weekly cadence. Only when there's something worth saying.
No spam. Unsubscribe any time.