IaC Drift Detection: Testing Terraform State vs Live Infrastructure
Infrastructure drift is the silent killer of reliable deployments. Your Terraform state says one thing; your actual cloud resources say another. Left unchecked, drift accumulates until a deployment fails in production at the worst possible moment.
This post covers how to systematically test for and detect IaC drift — the gap between what your state file declares and what's actually running in your cloud account.
What Is Infrastructure Drift?
Drift occurs when live infrastructure diverges from its declared state. Common causes:
- Manual changes made directly in the AWS/GCP/Azure console
- Automated systems (autoscalers, AWS Config remediations) modifying resources
- Partial applies where some resources updated but others failed
- External processes (security tools, cost optimizers) modifying tags or settings
- Cloud provider-side changes (automatic upgrades, endpoint migrations)
A Terraform plan will show drift as "changes to be applied," but only if you run it. Drift detection turns this reactive check into a proactive, scheduled test.
The Terraform Refresh Approach
The simplest drift detection uses terraform refresh and terraform plan together:
#!/bin/bash
# drift-check.sh
set -e
cd infra/
# Refresh state against real resources
terraform refresh -var-file=prod.tfvars
# Generate plan; exit code 2 means changes detected
terraform plan \
-var-file=prod.tfvars \
-detailed-exitcode \
-out=drift.tfplan
EXIT_CODE=$?
if [ $EXIT_CODE -eq 0 ]; then
echo "✅ No drift detected"
elif [ $EXIT_CODE -eq 2 ]; then
echo "⚠️ DRIFT DETECTED — review drift.tfplan"
terraform show drift.tfplan
exit 1
else
echo "❌ Terraform plan failed"
exit 1
fiThe -detailed-exitcode flag is key: exit 0 means no changes, exit 2 means changes exist, exit 1 means error.
Driftctl: Purpose-Built Drift Detection
driftctl is a dedicated tool that goes beyond what terraform plan shows. It detects:
- Resources in your cloud account that Terraform doesn't manage at all
- Resources Terraform manages but whose current state differs from plan
# Install
curl -L https://github.com/snyk/driftctl/releases/latest/download/driftctl_linux_amd64 \
-o /usr/local/bin/driftctl && chmod +x /usr/local/bin/driftctl
# Scan against Terraform state
driftctl scan \
--from tfstate://terraform.tfstate \
--to aws+tf
# Scan against remote state (S3)
driftctl scan \
--from tfstate+s3://my-tfstate-bucket/prod/terraform.tfstate \
--to aws+tfOutput shows three categories:
- Managed resources: declared and matching
- Drifted resources: declared but configuration has changed
- Unmanaged resources: exist in cloud but not in Terraform
Filtering False Positives
Some resources are intentionally managed outside Terraform. Create a .driftignore file:
# .driftignore
# ECS task definitions managed by deployment pipeline
aws_ecs_task_definition.*
# Lambda versions — old ones pile up
aws_lambda_function.*.qualified_arn
# Auto-scaling created ENIs
aws_network_interface.eni-*Writing Drift Tests with Terratest
For teams already using Terratest, you can add drift assertions directly to your Go test suite:
package test
import (
"testing"
"os/exec"
"strings"
"github.com/gruntwork-io/terratest/modules/terraform"
"github.com/stretchr/testify/assert"
)
func TestNoDriftInProduction(t *testing.T) {
t.Parallel()
terraformOptions := &terraform.Options{
TerraformDir: "../../infra/prod",
VarFiles: []string{"prod.tfvars"},
// Don't apply changes, just plan
PlanFilePath: "/tmp/drift-test.tfplan",
}
// Run plan with detailed exit code
planOutput := terraform.InitAndPlan(t, terraformOptions)
// Assert no changes are planned
resourceCount := terraform.GetResourceCount(t, planOutput)
assert.Equal(t, 0, resourceCount.Add,
"Resources to add — unexpected drift")
assert.Equal(t, 0, resourceCount.Change,
"Resources to change — configuration drift detected")
assert.Equal(t, 0, resourceCount.Destroy,
"Resources to destroy — unexpected drift")
}For more targeted drift checks on critical resources:
func TestCriticalResourcesDriftFree(t *testing.T) {
// Check specific high-value resources
criticalResources := []string{
"aws_rds_instance.primary",
"aws_elasticache_cluster.session_cache",
"aws_iam_role.app_execution_role",
}
terraformOptions := &terraform.Options{
TerraformDir: "../../infra/prod",
VarFiles: []string{"prod.tfvars"},
}
plan := terraform.InitAndPlanAndShow(t, terraformOptions)
for _, resource := range criticalResources {
changes := extractResourceChanges(plan, resource)
assert.Empty(t, changes,
"Critical resource %s has drift: %v", resource, changes)
}
}
func extractResourceChanges(plan string, resource string) []string {
// Parse plan output for specific resource changes
var changes []string
lines := strings.Split(plan, "\n")
inResource := false
for _, line := range lines {
if strings.Contains(line, resource) {
inResource = true
}
if inResource && strings.Contains(line, "~") {
changes = append(changes, strings.TrimSpace(line))
}
if inResource && line == "" {
inResource = false
}
}
return changes
}Scheduled Drift Detection in CI
Drift checks should run on a schedule, not just on PRs. A nightly GitHub Actions workflow:
# .github/workflows/drift-detection.yml
name: Infrastructure Drift Detection
on:
schedule:
- cron: '0 6 * * *' # 6 AM UTC daily
workflow_dispatch:
jobs:
detect-drift:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.7.x
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.DRIFT_CHECK_ROLE_ARN }}
aws-region: us-east-1
- name: Initialize Terraform
run: terraform -chdir=infra/prod init
- name: Check for drift
id: drift-check
run: |
terraform -chdir=infra/prod plan \
-var-file=prod.tfvars \
-detailed-exitcode \
-out=/tmp/drift.tfplan \
2>&1 | tee /tmp/plan-output.txt
EXIT_CODE=${PIPESTATUS[0]}
echo "exit_code=$EXIT_CODE" >> $GITHUB_OUTPUT
if [ $EXIT_CODE -eq 2 ]; then
echo "drift_detected=true" >> $GITHUB_OUTPUT
fi
continue-on-error: true
- name: Create drift report
if: steps.drift-check.outputs.drift_detected == 'true'
run: |
terraform -chdir=infra/prod show -no-color /tmp/drift.tfplan \
> /tmp/drift-report.txt
- name: Notify on drift
if: steps.drift-check.outputs.drift_detected == 'true'
uses: slackapi/slack-github-action@v1
with:
channel-id: '#infra-alerts'
payload: |
{
"text": "⚠️ Infrastructure drift detected in production",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "*Infrastructure Drift Detected*\n\nTerraform plan shows unplanned changes in production. Review required before next deployment.\n\n<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View Details>"
}
}
]
}
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}Drift Detection for Multi-Environment Setups
When managing multiple environments, drift in staging can predict production problems. Test cross-environment consistency:
#!/bin/bash
# check-env-drift.sh — compare staging vs production configuration
ENVS=("staging" "production")
DRIFT_FOUND=0
for ENV in "${ENVS[@]}"; do
echo "Checking $ENV..."
cd "infra/$ENV"
terraform init -backend-config="$ENV.backend.hcl" -reconfigure -input=false
PLAN_OUTPUT=$(terraform plan \
-var-file="$ENV.tfvars" \
-detailed-exitcode \
-no-color 2>&1)
EXIT_CODE=$?
if [ $EXIT_CODE -eq 2 ]; then
echo "⚠️ Drift in $ENV:"
echo "$PLAN_OUTPUT" | grep -E "^\s+(~|+|-)" | head -20
DRIFT_FOUND=1
fi
cd ../..
done
exit $DRIFT_FOUNDDrift Remediation Strategy
Detecting drift is only half the battle. You need a policy for what to do when drift is found:
Option 1: Import and adopt — if the manual change is intentional and correct, import the resource into Terraform state:
terraform import aws_security_group.app sg-0abc123def456Option 2: Revert to declared state — if the change was unauthorized, apply Terraform to restore:
terraform apply -target=aws_security_group.app -var-file=prod.tfvarsOption 3: Update the declaration — if the drift reveals that the Terraform code is wrong, update the .tf files to match reality, then run a plan to confirm zero changes.
Document your remediation in your runbook:
## Drift Remediation Runbook
1. Identify drifted resource from drift report
2. Determine root cause (manual change, automated system, etc.)
3. Classify: Intentional drift (update IaC) or Unauthorized drift (revert)
4. Create PR with fix
5. After merge, verify drift check passes
6. Post-mortem if production driftPreventing Drift Before It Starts
Drift detection is the safety net. Prevention is the goal:
- Lock down console access: Use SCPs (AWS) or org policies (GCP) to restrict direct resource modifications in production
- Enforce Terraform-only changes: Require all infrastructure changes via PR and CI/CD pipeline
- Tag resources with source:
managed_by = "terraform",module = "networking/vpc"— make manual changes visible - Use Sentinel or OPA policies: Block applies that would create untagged resources (which then drift undetected)
Adding HelpMeTest to Your Drift Workflow
Infrastructure drift often causes application-level symptoms before ops teams notice it. A security group change might break your app's database connection. A load balancer config change might cause 502s.
With HelpMeTest, you can run continuous end-to-end tests against your production environment that would catch drift-induced failures the moment they occur — before your drift detection job even runs:
Drift in aws_security_group.db_access
→ DB connections fail
→ HelpMeTest detects "Could not connect to database" in checkout flow
→ Alert fires in 5 minutes
→ Drift detection job confirms root cause 6 hours later at 6 AMThe combination of application-level monitoring (HelpMeTest) and IaC-level drift detection gives you fast alerting plus root cause attribution.
Summary
| Method | Speed | Depth | Best For |
|---|---|---|---|
terraform plan |
Fast | Managed resources only | PR checks |
terraform refresh + plan |
Moderate | Managed resources | Nightly CI |
| driftctl | Slow | Managed + unmanaged | Weekly audits |
| Terratest drift tests | Fast | Targeted resources | Critical paths |
Start with nightly terraform plan -detailed-exitcode in CI. Add driftctl for quarterly unmanaged resource audits. Graduate to Terratest drift tests for your most critical resources. And pair all of it with application-level monitoring to catch the symptoms before the cause.