HashiCorp Vault in Production: Secrets Management for Kubernetes

Kubernetes Secrets are base64 encoded, not encrypted. Here's how to set up Vault properly — Kubernetes auth, agent injection, secret rotation without restarts, and the audit trail regulated environments require.

Why Kubernetes secrets aren't enough

Kubernetes Secrets are base64 encoded, not encrypted. Anyone with read access to the etcd database or sufficient RBAC permissions can read every secret in your cluster in plaintext. For many organisations this is an acceptable risk with proper RBAC. For anything handling financial data, PII, or operating in a regulated environment, it isn't.

HashiCorp Vault solves the problem properly — secrets are encrypted at rest and in transit, access is audited, secrets can be rotated without application restarts, and you get a complete audit log of who accessed what and when. Here's how to set it up and integrate it with Kubernetes in a production-viable way.

Vault architecture for Kubernetes

The integration works through the Vault Agent Injector — a mutating webhook that intercepts pod creation and injects a Vault Agent sidecar. The sidecar authenticates to Vault using the pod's Kubernetes service account, fetches the required secrets, and writes them to a shared in-memory volume. Your application reads secrets from files, not environment variables.

# Install Vault via Helm
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

helm install vault hashicorp/vault   --namespace vault   --create-namespace   --set "server.ha.enabled=true"   --set "server.ha.replicas=3"   --set "injector.enabled=true"

# Initialize Vault (first time only)
kubectl exec -n vault vault-0 -- vault operator init   -key-shares=5   -key-threshold=3   -format=json > vault-init.json

Critical: Store the unseal keys and root token from vault-init.json somewhere secure immediately — AWS Secrets Manager, a password manager, encrypted storage. If you lose these, you lose access to Vault permanently. Delete the local file after storing safely.

Configuring Kubernetes authentication

# Enable Kubernetes auth method
vault auth enable kubernetes

# Configure it with the cluster's API server
vault write auth/kubernetes/config   kubernetes_host="https://$(kubectl get svc kubernetes -o jsonpath='{.spec.clusterIP}'):443"   kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt   token_reviewer_jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token

# Create a policy for your application
vault policy write api-service - <<EOF
path "secret/data/api-service/*" {{
  capabilities = ["read"]
}}
EOF

# Create a role binding service account to policy
vault write auth/kubernetes/role/api-service   bound_service_account_names=api-service   bound_service_account_namespaces=production   policies=api-service   ttl=1h

Writing and reading secrets

# Write a secret to Vault
vault kv put secret/api-service/database   host="db.production.internal"   username="api_user"   password="$(openssl rand -base64 32)"

# Read it back to verify
vault kv get secret/api-service/database

Injecting secrets into pods

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
  namespace: production
spec:
  template:
    metadata:
      annotations:
        # These annotations tell the injector what to fetch
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "api-service"
        vault.hashicorp.com/agent-inject-secret-database.env: "secret/data/api-service/database"
        vault.hashicorp.com/agent-inject-template-database.env: |
          {{`{{- with secret "secret/data/api-service/database" -}}
          export DB_HOST="{{ .Data.data.host }}"
          export DB_USER="{{ .Data.data.username }}"
          export DB_PASS="{{ .Data.data.password }}"
          {{- end }}`}}
    spec:
      serviceAccountName: api-service
      containers:
      - name: api
        image: api-service:latest
        command: ["/bin/sh", "-c"]
        args: ["source /vault/secrets/database.env && ./api-server"]

Secret rotation without restarts

One of Vault's biggest operational advantages is that secrets can be rotated without application restarts. The Vault Agent sidecar re-fetches secrets on the configured TTL and rewrites the secrets file. If your application reads from the file on each request rather than caching at startup, rotation is completely transparent.

# Rotate a secret — no pod restarts needed
vault kv put secret/api-service/database   host="db.production.internal"   username="api_user"   password="$(openssl rand -base64 32)"

# Vault Agent picks up the new value within the TTL window
# Application reads fresh value on next file read

Audit logging

# Enable file audit log
vault audit enable file file_path=/vault/logs/audit.log

# Every read, write, and auth event is logged
# Who accessed which secret, when, from which pod
# This is the compliance audit trail most regulated environments require

Worth knowing: The audit log is append-only and cannot be disabled once Vault has processed a request with it enabled. This is intentional — it means even a compromised Vault administrator cannot retroactively hide access events.