Infrastructure Drift Detection and Remediation

Infrastructure drift is the gap between what your code says your infrastructure should look like and what it actually looks like. A security group rule added through the console, a tag changed manually, a scaling parameter tuned during an incident and never committed — these are all drift. Left unchecked, drift erodes the value of infrastructure as code because the code no longer reflects reality.

Why Drift Happens

Drift has three common causes:

Manual changes. Someone logs into the AWS console during an incident, opens a port, and forgets to update the Terraform code afterward. The fix worked, so nobody circles back.

Out-of-band automation. A script or Lambda function modifies resources that Terraform also manages. Auto Scaling groups are a frequent offender: AWS adjusts the desired count, which conflicts with the Terraform-declared value.

State staleness. Terraform state gets out of sync when multiple people run applies concurrently without proper locking, or when state is restored from an old backup.

Detecting Drift with Terraform

The simplest drift detection is terraform plan run against your current state. If the plan shows changes you did not expect, you have drift:

terraform plan -detailed-exitcode -out=plan.tfplan
# Exit code 0: no changes
# Exit code 1: error
# Exit code 2: changes detected

The -detailed-exitcode flag lets you script around the result. Exit code 2 means the real infrastructure differs from the code.

To get a machine-readable output:

terraform plan -json -out=plan.tfplan > plan.json
terraform show -json plan.tfplan | \
  jq '.resource_changes[] | select(.change.actions != ["no-op"]) | {address, actions: .change.actions}'

This produces a list of every resource that has drifted, along with the action Terraform would take to reconcile it.

Automated Drift Detection in CI

Run drift detection on a schedule. A GitHub Actions workflow that checks every six hours:

name: Drift Detection
on:
  schedule:
    - cron: '0 */6 * * *'
  workflow_dispatch:

jobs:
  detect-drift:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        environment: [staging, production]
    steps:
      - uses: actions/checkout@v4

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.9.0

      - name: Terraform Init
        working-directory: environments/${{ matrix.environment }}
        run: terraform init -backend-config=backend.hcl

      - name: Detect Drift
        id: plan
        working-directory: environments/${{ matrix.environment }}
        run: |
          terraform plan -detailed-exitcode -no-color \
            -out=drift.tfplan 2>&1 | tee plan_output.txt
          echo "exitcode=$?" >> $GITHUB_OUTPUT
        continue-on-error: true

      - name: Report Drift
        if: steps.plan.outputs.exitcode == '2'
        run: |
          echo "::warning::Drift detected in ${{ matrix.environment }}"
          # Post to Slack
          curl -X POST "${{ secrets.SLACK_WEBHOOK }}" \
            -H 'Content-Type: application/json' \
            -d "{\"text\":\"Infrastructure drift detected in ${{ matrix.environment }}. Review the plan output.\"}"

      - name: Upload Plan
        if: steps.plan.outputs.exitcode == '2'
        uses: actions/upload-artifact@v4
        with:
          name: drift-plan-${{ matrix.environment }}
          path: environments/${{ matrix.environment }}/plan_output.txt

When drift is detected, the workflow alerts the team via Slack and uploads the plan output as an artifact for review.

Remediation Strategies

When you find drift, you have two choices:

Option 1: Reconcile (Apply the Code)

If the code is correct and the manual change should be reverted:

terraform apply plan.tfplan

This brings the infrastructure back in line with the code. Use this when someone made an unauthorized change or a temporary fix that should be undone.

Option 2: Update the Code

If the manual change was intentional and should be kept, update the Terraform code to match reality:

# Import the current state of a manually created resource
terraform import aws_security_group.web sg-0abc123def456

# Or refresh state and update code to match
terraform refresh
terraform plan  # Review what Terraform wants to change
# Update .tf files to match the desired state

A more structured approach uses terraform state show to inspect the current state and update the code accordingly:

terraform state show aws_security_group.web
# Copy the relevant attributes into your .tf file

Preventing Drift

Detection is reactive. Prevention is better.

Lock Down Console Access

Use IAM policies that grant read-only console access for production accounts. Changes should go through Terraform, not the console:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Action": [
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:RevokeSecurityGroupIngress",
        "ec2:ModifyInstanceAttribute"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:PrincipalTag/role": "terraform-automation"
        }
      }
    }
  ]
}

This policy denies security group modifications unless the principal is the Terraform automation role.

Use Lifecycle Rules

For resources where drift is expected (like Auto Scaling group desired counts), tell Terraform to ignore those attributes:

resource "aws_autoscaling_group" "web" {
  desired_capacity = 3
  min_size         = 2
  max_size         = 20

  lifecycle {
    ignore_changes = [desired_capacity]
  }
}

This prevents Terraform from reporting drift on the desired count while still managing the other attributes.

Policy as Code with OPA

Use Open Policy Agent to enforce rules about what Terraform plans are allowed to apply:

package terraform

deny[msg] {
  resource := input.resource_changes[_]
  resource.type == "aws_security_group_rule"
  resource.change.after.cidr_blocks[_] == "0.0.0.0/0"
  resource.change.after.from_port == 22
  msg := "SSH must not be open to 0.0.0.0/0"
}

Run the policy check in CI before terraform apply to catch policy violations regardless of whether they were introduced manually or through code.

Building a Drift Report

For organizations managing many Terraform workspaces, build a dashboard that aggregates drift status:

#!/bin/bash
# drift-report.sh
for dir in environments/*/; do
  env=$(basename "$dir")
  cd "$dir"
  terraform init -backend-config=backend.hcl > /dev/null 2>&1
  terraform plan -detailed-exitcode > /dev/null 2>&1
  exitcode=$?
  if [ $exitcode -eq 2 ]; then
    echo "$env: DRIFT DETECTED"
  elif [ $exitcode -eq 0 ]; then
    echo "$env: IN SYNC"
  else
    echo "$env: ERROR"
  fi
  cd - > /dev/null
done

Feed this into a monitoring system so drift becomes a tracked metric rather than a surprise.

Wrap-up

Infrastructure drift is inevitable in any environment where humans have access to the cloud console. The defense is layered: detect drift automatically with scheduled plans, remediate by either applying the code or updating it to match reality, and prevent future drift by restricting manual access and using lifecycle rules for expected variations. Treat drift like a bug: track it, fix it, and build systems that prevent it from recurring.