HashiCorp Vault for Secrets Management

The Problem with Secrets

Secrets end up everywhere: environment variables in deployment scripts, .env files committed to Git, encrypted blobs in S3, sticky notes on monitors. Each approach has the same flaw: no audit trail, no rotation, and no fine-grained access control.

HashiCorp Vault solves this by providing a centralized, audited, policy-controlled system for managing secrets. Applications authenticate to Vault, request the secrets they need, and Vault logs every access. When a credential is compromised, you rotate it in one place instead of hunting through thirty repositories.

Core Concepts

Secrets engines are backends that store or generate secrets. The KV (key-value) engine stores static secrets. The database engine generates short-lived database credentials on demand. The PKI engine issues TLS certificates. Each engine mounts at a path in Vault’s API.

Auth methods determine how clients prove their identity. Kubernetes pods use the Kubernetes auth method. CI/CD pipelines use AppRole. Humans use OIDC or LDAP. Each method maps an external identity to a Vault policy.

Policies define what a client can do. They are written in HCL and grant or deny access to specific paths. A policy is the boundary between “this service can read database credentials” and “this service can also read the payment gateway API key.”

Setting Up Vault

For development, run Vault in dev mode:

vault server -dev -dev-root-token-id="dev-root-token"

In another terminal, configure the CLI:

export VAULT_ADDR="http://127.0.0.1:8200"
export VAULT_TOKEN="dev-root-token"

vault status

For production, you deploy Vault with a proper storage backend (Raft, Consul, or cloud-managed), TLS, and auto-unseal using a cloud KMS. Dev mode is in-memory and should never be used for real workloads.

KV Secrets Engine

The KV engine stores arbitrary key-value pairs. Version 2 (the default) keeps a history of changes:

# Enable KV v2 at the default path
vault secrets enable -path=secret kv-v2

# Write a secret
vault kv put secret/api/payments \
  stripe_key="sk_live_abc123" \
  stripe_webhook_secret="whsec_xyz789"

# Read a secret
vault kv get secret/api/payments

# Read a specific field
vault kv get -field=stripe_key secret/api/payments

# Read a previous version
vault kv get -version=1 secret/api/payments

Using the HTTP API from an application:

curl -s \
  -H "X-Vault-Token: $VAULT_TOKEN" \
  "$VAULT_ADDR/v1/secret/data/api/payments" | jq '.data.data'

{
  "stripe_key": "sk_live_abc123",
  "stripe_webhook_secret": "whsec_xyz789"
}

Policies

Policies control access with path-based rules:

# policies/api-service.hcl
path "secret/data/api/payments" {
  capabilities = ["read"]
}

path "secret/data/api/shared/*" {
  capabilities = ["read", "list"]
}

# Deny access to admin secrets
path "secret/data/admin/*" {
  capabilities = ["deny"]
}

Apply the policy:

vault policy write api-service policies/api-service.hcl

The capabilities are create, read, update, delete, list, sudo, and deny. Deny always wins. A service with the api-service policy can read payment secrets but cannot access admin secrets, even if another attached policy grants broader access.

Auth Methods

AppRole for Applications

AppRole is designed for machine-to-machine authentication. It uses a role ID (like a username) and a secret ID (like a password):

# Enable AppRole
vault auth enable approle

# Create a role
vault write auth/approle/role/api-service \
  token_policies="api-service" \
  token_ttl=1h \
  token_max_ttl=4h \
  secret_id_ttl=24h \
  secret_id_num_uses=10

# Get the role ID (stable, can be embedded in config)
vault read auth/approle/role/api-service/role-id

# Generate a secret ID (ephemeral, delivered securely)
vault write -f auth/approle/role/api-service/secret-id

The application authenticates by exchanging the role ID and secret ID for a token:

vault write auth/approle/login \
  role_id="$ROLE_ID" \
  secret_id="$SECRET_ID"

Kubernetes Auth

For pods running in Kubernetes, Vault can verify the pod’s service account token:

vault auth enable kubernetes

vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc"

vault write auth/kubernetes/role/api-service \
  bound_service_account_names=api-service \
  bound_service_account_namespaces=production \
  policies=api-service \
  ttl=1h

The pod authenticates using its service account JWT. No hardcoded credentials needed. The Vault Agent sidecar or the CSI provider can automate this entirely.

Dynamic Secrets

Dynamic secrets are generated on demand and automatically revoked after their TTL expires. This is Vault’s most powerful feature.

Database Credentials

# Enable the database engine
vault secrets enable database

# Configure a PostgreSQL connection
vault write database/config/myapp-db \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/myapp?sslmode=require" \
  allowed_roles="api-readonly,api-readwrite" \
  username="vault_admin" \
  password="vault_admin_password"

# Create a role that generates read-only credentials
vault write database/roles/api-readonly \
  db_name=myapp-db \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
    GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  revocation_statements="DROP ROLE IF EXISTS \"{{name}}\";" \
  default_ttl=1h \
  max_ttl=24h

Now when a service requests credentials:

vault read database/creds/api-readonly

Vault creates a new PostgreSQL user with a random password, returns the credentials, and automatically drops the user when the TTL expires. If credentials are compromised, they stop working in at most one hour. No manual rotation needed.

Vault Agent

Vault Agent runs as a sidecar or daemon that handles authentication and secret retrieval automatically. It can render secrets into template files that your application reads:

# vault-agent-config.hcl
auto_auth {
  method "kubernetes" {
    mount_path = "auth/kubernetes"
    config = {
      role = "api-service"
    }
  }

  sink "file" {
    config = {
      path = "/tmp/vault-token"
    }
  }
}

template {
  source      = "/etc/vault/templates/env.tmpl"
  destination = "/app/config/.env"
  perms       = 0600
}

# /etc/vault/templates/env.tmpl
{{ with secret "secret/data/api/payments" }}
STRIPE_KEY={{ .Data.data.stripe_key }}
STRIPE_WEBHOOK_SECRET={{ .Data.data.stripe_webhook_secret }}
{{ end }}

{{ with secret "database/creds/api-readonly" }}
DATABASE_URL=postgresql://{{ .Data.username }}:{{ .Data.password }}@db.internal:5432/myapp
{{ end }}

The agent authenticates to Vault, fetches secrets, renders the .env file, and re-renders it when secrets rotate. Your application just reads a file. It does not need to know Vault exists.

Audit Logging

Enable audit logging to track every secret access:

vault audit enable file file_path=/var/log/vault/audit.log

Every request and response is logged as JSON, including who accessed which path, when, and whether the request was allowed or denied. This is essential for compliance (SOC 2, PCI DSS, HIPAA) and incident investigation.

Production Considerations

High availability. Run Vault in HA mode with Raft storage and at least three nodes. A single Vault server is a single point of failure for your entire secret infrastructure.

Auto-unseal. Use AWS KMS, GCP Cloud KMS, or Azure Key Vault to auto-unseal Vault on restart. Manual unsealing with Shamir keys is operationally painful.

Backup. Vault’s Raft storage supports snapshots. Take automated snapshots and store them encrypted in a separate location.

Lease management. Monitor the number of active leases. Applications that request dynamic secrets without renewing or revoking them create lease buildup that can degrade Vault performance.

Start by migrating one set of secrets (database credentials for a staging environment, for example) from environment variables to Vault. Once the workflow is proven, expand to production and add dynamic secrets. The incremental approach lets you build confidence without a big-bang migration.