AWS Secrets Manager Tutorial: Storing and Rotating Secrets

AWS Secrets Manager stores sensitive strings — database passwords, API keys, OAuth tokens — encrypted at rest with KMS, retrieved over IAM-authenticated APIs, and optionally rotated automatically. It costs more than SSM Parameter Store but earns its keep when you need rotation, cross-account sharing, or versioned secrets with staging labels.

What and Why

A secret in Secrets Manager is just a JSON blob (or plain string) wrapped in a managed envelope: a name, a KMS key, a rotation policy, and a set of versions tagged with staging labels like AWSCURRENT and AWSPREVIOUS. Your app reads the secret by name and always gets the current version unless you ask otherwise.

Why not just use environment variables? Env vars get committed by accident, leak in logs, and never rotate. Why not Parameter Store SecureString? It is cheaper and great for config, but it lacks first-class rotation, cross-region replication, and the staging-label workflow that makes zero-downtime credential changes possible.

Pick Secrets Manager when secrets must rotate on a schedule, when multiple services share a database password that may change, or when you need cross-account access via resource policies.

Mental Model

Each secret has a stable name and an unbounded version history. AWS labels exactly one version as AWSCURRENT. Rotation creates a new version, tests it, then atomically moves the AWSCURRENT label. The previous version keeps AWSPREVIOUS for one cycle so in-flight requests don’t break. Your code always asks for AWSCURRENT.

secret: prod/db/password
  v1  [AWSPREVIOUS]
  v2  [AWSCURRENT]
  v3  [AWSPENDING] (during rotation)

When rotation finishes successfully, v3 becomes AWSCURRENT, v2 becomes AWSPREVIOUS, and v1 loses its label.

Hands-on Example

Create a secret holding a database password, fetch it from an app, and enable rotation.

aws secretsmanager create-secret \
  --name prod/orders/db \
  --description "Orders RDS master password" \
  --secret-string '{"username":"orders","password":"S3edChange!"}'

In a Node.js app, fetch it on startup:

import { SecretsManagerClient, GetSecretValueCommand } from "@aws-sdk/client-secrets-manager";
const sm = new SecretsManagerClient({});
const res = await sm.send(new GetSecretValueCommand({ SecretId: "prod/orders/db" }));
const { username, password } = JSON.parse(res.SecretString);

Attach an IAM policy to the app role granting only secretsmanager:GetSecretValue on that exact ARN, plus kms:Decrypt on the KMS key.

Lambda rotator + Secrets Manager + RDS

Step 1 createSecret:   new password generated -> AWSPENDING
Step 2 setSecret:      ALTER USER on RDS to new password
Step 3 testSecret:     connect using AWSPENDING -> ok?
Step 4 finishSecret:   move AWSCURRENT label to new version

     v(n)  [AWSCURRENT]  -->  [AWSPREVIOUS]
     v(n+1)[AWSPENDING] -->  [AWSCURRENT]

Enable rotation with a managed Lambda template:

aws secretsmanager rotate-secret \
  --secret-id prod/orders/db \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:123:function:SecretsManagerRDSRotation \
  --rotation-rules AutomaticallyAfterDays=30

The first rotation runs immediately. From then on, your password silently rolls every 30 days and your app picks it up on the next cache miss.

Common Pitfalls

• Caching forever. Apps that fetch the secret at boot and never refresh will break the moment rotation runs. Use the AWS Secrets Manager client-side caching library or a TTL of a few minutes.
• Wildcard IAM resources. Resource: "*" on GetSecretValue is an audit finding waiting to happen. Pin to specific ARNs.
• Forgotten KMS permissions. A correct Secrets Manager policy still fails if the role cannot kms:Decrypt the underlying key — especially when using customer-managed keys.
• Rotating without testing. The default RDS rotation Lambda only works for the master user. Custom users or non-RDS targets need a custom Lambda; test it in staging first.
• Cost surprise. Secrets cost USD 0.40 per secret per month plus API calls. A microservice fleet hitting one secret thousands of times per minute racks up bills fast — cache aggressively.

Production Tips

Use a hierarchical naming convention like env/service/purpose so policies and dashboards can wildcard cleanly per environment. Replicate critical secrets to a DR region with replicate-secret-to-regions. Tag every secret with owner and service so cost reports stay readable.

Wire CloudTrail events for GetSecretValue into your SIEM — sudden access from unexpected principals is a red flag. For local dev, never hard-code; use the same SDK call with a dev secret and IAM role assumed by your laptop.

Wrap-up

Secrets Manager is the right home for credentials that need to rotate, version, or cross account boundaries. Create the secret, lock the IAM and KMS policies, cache reads in your app, and let scheduled rotation handle the rest. The extra cost over Parameter Store buys you a real rotation story and a much cleaner incident timeline.