IaC Drift Detection: Testing Terraform State vs Live Infrastructure

IaC Drift Detection: Testing Terraform State vs Live Infrastructure

Infrastructure drift is the silent killer of reliable deployments. Your Terraform state says one thing; your actual cloud resources say another. Left unchecked, drift accumulates until a deployment fails in production at the worst possible moment.

This post covers how to systematically test for and detect IaC drift — the gap between what your state file declares and what's actually running in your cloud account.

What Is Infrastructure Drift?

Drift occurs when live infrastructure diverges from its declared state. Common causes:

  • Manual changes made directly in the AWS/GCP/Azure console
  • Automated systems (autoscalers, AWS Config remediations) modifying resources
  • Partial applies where some resources updated but others failed
  • External processes (security tools, cost optimizers) modifying tags or settings
  • Cloud provider-side changes (automatic upgrades, endpoint migrations)

A Terraform plan will show drift as "changes to be applied," but only if you run it. Drift detection turns this reactive check into a proactive, scheduled test.

The Terraform Refresh Approach

The simplest drift detection uses terraform refresh and terraform plan together:

#!/bin/bash
# drift-check.sh

set -e

cd infra/

# Refresh state against real resources
terraform refresh -var-file=prod.tfvars

# Generate plan; exit code 2 means changes detected
terraform plan \
  -var-file=prod.tfvars \
  -detailed-exitcode \
  -out=drift.tfplan

EXIT_CODE=$?

if [ $EXIT_CODE -eq 0 ]; then
  echo "✅ No drift detected"
elif [ $EXIT_CODE -eq 2 ]; then
  echo "⚠️ DRIFT DETECTED — review drift.tfplan"
  terraform show drift.tfplan
  exit 1
else
  echo "❌ Terraform plan failed"
  exit 1
fi

The -detailed-exitcode flag is key: exit 0 means no changes, exit 2 means changes exist, exit 1 means error.

Driftctl: Purpose-Built Drift Detection

driftctl is a dedicated tool that goes beyond what terraform plan shows. It detects:

  • Resources in your cloud account that Terraform doesn't manage at all
  • Resources Terraform manages but whose current state differs from plan
# Install
curl -L https://github.com/snyk/driftctl/releases/latest/download/driftctl_linux_amd64 \
  -o /usr/local/bin/driftctl && chmod +x /usr/local/bin/driftctl

# Scan against Terraform state
driftctl scan \
  --from tfstate://terraform.tfstate \
  --to aws+tf

# Scan against remote state (S3)
driftctl scan \
  --from tfstate+s3://my-tfstate-bucket/prod/terraform.tfstate \
  --to aws+tf

Output shows three categories:

  • Managed resources: declared and matching
  • Drifted resources: declared but configuration has changed
  • Unmanaged resources: exist in cloud but not in Terraform

Filtering False Positives

Some resources are intentionally managed outside Terraform. Create a .driftignore file:

# .driftignore
# ECS task definitions managed by deployment pipeline
aws_ecs_task_definition.*

# Lambda versions — old ones pile up
aws_lambda_function.*.qualified_arn

# Auto-scaling created ENIs
aws_network_interface.eni-*

Writing Drift Tests with Terratest

For teams already using Terratest, you can add drift assertions directly to your Go test suite:

package test

import (
    "testing"
    "os/exec"
    "strings"
    
    "github.com/gruntwork-io/terratest/modules/terraform"
    "github.com/stretchr/testify/assert"
)

func TestNoDriftInProduction(t *testing.T) {
    t.Parallel()
    
    terraformOptions := &terraform.Options{
        TerraformDir: "../../infra/prod",
        VarFiles:     []string{"prod.tfvars"},
        // Don't apply changes, just plan
        PlanFilePath: "/tmp/drift-test.tfplan",
    }
    
    // Run plan with detailed exit code
    planOutput := terraform.InitAndPlan(t, terraformOptions)
    
    // Assert no changes are planned
    resourceCount := terraform.GetResourceCount(t, planOutput)
    
    assert.Equal(t, 0, resourceCount.Add,
        "Resources to add — unexpected drift")
    assert.Equal(t, 0, resourceCount.Change,
        "Resources to change — configuration drift detected")
    assert.Equal(t, 0, resourceCount.Destroy,
        "Resources to destroy — unexpected drift")
}

For more targeted drift checks on critical resources:

func TestCriticalResourcesDriftFree(t *testing.T) {
    // Check specific high-value resources
    criticalResources := []string{
        "aws_rds_instance.primary",
        "aws_elasticache_cluster.session_cache",
        "aws_iam_role.app_execution_role",
    }
    
    terraformOptions := &terraform.Options{
        TerraformDir: "../../infra/prod",
        VarFiles:     []string{"prod.tfvars"},
    }
    
    plan := terraform.InitAndPlanAndShow(t, terraformOptions)
    
    for _, resource := range criticalResources {
        changes := extractResourceChanges(plan, resource)
        assert.Empty(t, changes,
            "Critical resource %s has drift: %v", resource, changes)
    }
}

func extractResourceChanges(plan string, resource string) []string {
    // Parse plan output for specific resource changes
    var changes []string
    lines := strings.Split(plan, "\n")
    inResource := false
    
    for _, line := range lines {
        if strings.Contains(line, resource) {
            inResource = true
        }
        if inResource && strings.Contains(line, "~") {
            changes = append(changes, strings.TrimSpace(line))
        }
        if inResource && line == "" {
            inResource = false
        }
    }
    return changes
}

Scheduled Drift Detection in CI

Drift checks should run on a schedule, not just on PRs. A nightly GitHub Actions workflow:

# .github/workflows/drift-detection.yml
name: Infrastructure Drift Detection

on:
  schedule:
    - cron: '0 6 * * *'  # 6 AM UTC daily
  workflow_dispatch:

jobs:
  detect-drift:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.7.x
      
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.DRIFT_CHECK_ROLE_ARN }}
          aws-region: us-east-1
      
      - name: Initialize Terraform
        run: terraform -chdir=infra/prod init
      
      - name: Check for drift
        id: drift-check
        run: |
          terraform -chdir=infra/prod plan \
            -var-file=prod.tfvars \
            -detailed-exitcode \
            -out=/tmp/drift.tfplan \
            2>&1 | tee /tmp/plan-output.txt
          
          EXIT_CODE=${PIPESTATUS[0]}
          echo "exit_code=$EXIT_CODE" >> $GITHUB_OUTPUT
          
          if [ $EXIT_CODE -eq 2 ]; then
            echo "drift_detected=true" >> $GITHUB_OUTPUT
          fi
        continue-on-error: true
      
      - name: Create drift report
        if: steps.drift-check.outputs.drift_detected == 'true'
        run: |
          terraform -chdir=infra/prod show -no-color /tmp/drift.tfplan \
            > /tmp/drift-report.txt
      
      - name: Notify on drift
        if: steps.drift-check.outputs.drift_detected == 'true'
        uses: slackapi/slack-github-action@v1
        with:
          channel-id: '#infra-alerts'
          payload: |
            {
              "text": "⚠️ Infrastructure drift detected in production",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "*Infrastructure Drift Detected*\n\nTerraform plan shows unplanned changes in production. Review required before next deployment.\n\n<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View Details>"
                  }
                }
              ]
            }
        env:
          SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}

Drift Detection for Multi-Environment Setups

When managing multiple environments, drift in staging can predict production problems. Test cross-environment consistency:

#!/bin/bash
# check-env-drift.sh — compare staging vs production configuration

ENVS=("staging" "production")
DRIFT_FOUND=0

for ENV in "${ENVS[@]}"; do
    echo "Checking $ENV..."
    
    cd "infra/$ENV"
    terraform init -backend-config="$ENV.backend.hcl" -reconfigure -input=false
    
    PLAN_OUTPUT=$(terraform plan \
        -var-file="$ENV.tfvars" \
        -detailed-exitcode \
        -no-color 2>&1)
    
    EXIT_CODE=$?
    
    if [ $EXIT_CODE -eq 2 ]; then
        echo "⚠️ Drift in $ENV:"
        echo "$PLAN_OUTPUT" | grep -E "^\s+(~|+|-)" | head -20
        DRIFT_FOUND=1
    fi
    
    cd ../..
done

exit $DRIFT_FOUND

Drift Remediation Strategy

Detecting drift is only half the battle. You need a policy for what to do when drift is found:

Option 1: Import and adopt — if the manual change is intentional and correct, import the resource into Terraform state:

terraform import aws_security_group.app sg-0abc123def456

Option 2: Revert to declared state — if the change was unauthorized, apply Terraform to restore:

terraform apply -target=aws_security_group.app -var-file=prod.tfvars

Option 3: Update the declaration — if the drift reveals that the Terraform code is wrong, update the .tf files to match reality, then run a plan to confirm zero changes.

Document your remediation in your runbook:

## Drift Remediation Runbook

1. Identify drifted resource from drift report
2. Determine root cause (manual change, automated system, etc.)
3. Classify: Intentional drift (update IaC) or Unauthorized drift (revert)
4. Create PR with fix
5. After merge, verify drift check passes
6. Post-mortem if production drift

Preventing Drift Before It Starts

Drift detection is the safety net. Prevention is the goal:

  • Lock down console access: Use SCPs (AWS) or org policies (GCP) to restrict direct resource modifications in production
  • Enforce Terraform-only changes: Require all infrastructure changes via PR and CI/CD pipeline
  • Tag resources with source: managed_by = "terraform", module = "networking/vpc" — make manual changes visible
  • Use Sentinel or OPA policies: Block applies that would create untagged resources (which then drift undetected)

Adding HelpMeTest to Your Drift Workflow

Infrastructure drift often causes application-level symptoms before ops teams notice it. A security group change might break your app's database connection. A load balancer config change might cause 502s.

With HelpMeTest, you can run continuous end-to-end tests against your production environment that would catch drift-induced failures the moment they occur — before your drift detection job even runs:

Drift in aws_security_group.db_access
  → DB connections fail
  → HelpMeTest detects "Could not connect to database" in checkout flow
  → Alert fires in 5 minutes
  → Drift detection job confirms root cause 6 hours later at 6 AM

The combination of application-level monitoring (HelpMeTest) and IaC-level drift detection gives you fast alerting plus root cause attribution.

Summary

Method Speed Depth Best For
terraform plan Fast Managed resources only PR checks
terraform refresh + plan Moderate Managed resources Nightly CI
driftctl Slow Managed + unmanaged Weekly audits
Terratest drift tests Fast Targeted resources Critical paths

Start with nightly terraform plan -detailed-exitcode in CI. Add driftctl for quarterly unmanaged resource audits. Graduate to Terratest drift tests for your most critical resources. And pair all of it with application-level monitoring to catch the symptoms before the cause.

Read more

Start now free