Cloud Cost Testing: Preventing Expensive Mistakes Before Deployment

24 Aug 2026 — 7 min read

Every engineering team has a war story about cloud costs. A misconfigured NAT gateway left running over a weekend. An S3 bucket with lifecycle policies accidentally removed, slowly accumulating terabytes. A developer who switched from db.t3.micro to db.r6g.4xlarge "just to test" and forgot to revert. The bill arrives weeks later, the incident is reconstructed from memory, and a new runbook gets written.

The common thread: these mistakes were detectable before deployment. Cost testing — the practice of asserting cost implications as part of your CI pipeline — catches these issues at PR time, when they're cheap to fix.

Infracost: Cost Estimation in CI

Infracost compares the cost of your current infrastructure to the cost after a proposed Terraform change. It integrates with GitHub, GitLab, and Azure DevOps to post cost diffs directly on PRs.

Installation and Basic Usage

brew install infracost
infracost auth login

Generate a cost estimate:

infracost breakdown --path ./terraform

Output:

Name                                   Monthly Qty  Unit    Monthly Cost

aws_db_instance.main
├─ Database instance (db.t3.micro)             730  hours        $24.82
├─ Storage (gp2, 20 GB)                         20  GB            $2.30
└─ Backup storage                    Cost depends on usage

aws_instance.web (x3)
├─ Instance usage (t3.medium)                2,190  hours        $89.57
└─ root_volume: Storage (gp2, 30 GB)            90  GB           $10.35

OVERALL TOTAL                                                    $126.04

Compare with a diff (what the PR changes cost):

infracost diff --path ./terraform \
  --compare-to infracost-base.json \
  --format json > infracost-diff.json

infracost output --path infracost-diff.json --format table

GitHub Actions Integration

# .github/workflows/infracost.yml
name: Infracost
on: [pull_request]

jobs:
  infracost:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write

    steps:
      - uses: actions/checkout@v4

      - name: Setup Infracost
        uses: infracost/actions/setup@v3
        with:
          api-key: ${{ secrets.INFRACOST_API_KEY }}

      - name: Generate Infracost cost estimate baseline
        run: |
          infracost breakdown --path=terraform/ \
            --format=json \
            --out-file=/tmp/infracost-base.json
        env:
          TF_VAR_environment: production

      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.head.sha }}

      - name: Generate Infracost diff
        run: |
          infracost diff --path=terraform/ \
            --format=json \
            --compare-to=/tmp/infracost-base.json \
            --out-file=/tmp/infracost-diff.json
        env:
          TF_VAR_environment: production

      - name: Post Infracost comment
        run: |
          infracost comment github --path=/tmp/infracost-diff.json \
            --repo=$GITHUB_REPOSITORY \
            --github-token=${{ secrets.GITHUB_TOKEN }} \
            --pull-request=${{ github.event.pull_request.number }} \
            --behavior=update

This posts a comment on every PR showing exactly what changes will cost. A PR that changes db.t3.micro to db.r6g.4xlarge will show a monthly cost increase of ~$2,000 — impossible to miss.

Failing the Pipeline on Cost Increases

Beyond informational comments, you can fail the build if costs exceed a threshold:

- name: Check cost increase
  run: |
    MONTHLY_DIFF=$(cat /tmp/infracost-diff.json | \
      jq '.projects[].diff.totalMonthlyCost | tonumber' | \
      awk '{sum += $1} END {print sum}')
    echo "Monthly cost change: $${MONTHLY_DIFF}"

    # Fail if increase is more than $100/month
    if (( $(echo "$MONTHLY_DIFF > 100" | bc -l) )); then
      echo "ERROR: Monthly cost increase of $${MONTHLY_DIFF} exceeds $100 threshold"
      exit 1
    fi

For more granular thresholds per resource type:

# Check if any single resource costs more than $500/month
HIGH_COST_RESOURCES=$(cat /tmp/infracost-diff.json | jq '
  [.projects[].breakdown.resources[]
    | select(.monthlyCost != null)
    | select(.monthlyCost | tonumber > 500)
    | {name: .name, cost: .monthlyCost}]
')

if [ "$(echo $HIGH_COST_RESOURCES | jq length)" -gt 0 ]; then
  echo "ERROR: Resources exceeding $500/month:"
  echo $HIGH_COST_RESOURCES | jq .
  exit 1
fi

Testing for Cost Anomalies: S3 and NAT Gateways

Some resources are particularly prone to runaway costs. Test for these patterns explicitly.

S3 Bucket Policy Testing

An S3 bucket without lifecycle policies accumulates data indefinitely. Test for this in Terraform:

# policy/s3_cost.rego
package terraform.s3

# Every S3 bucket must have a lifecycle rule
deny[msg] {
  resource := input.resource_changes[_]
  resource.type == "aws_s3_bucket"
  resource.change.actions[_] == "create"

  # Check that a corresponding lifecycle configuration exists
  lifecycle_exists := count([r |
    r := input.resource_changes[_]
    r.type == "aws_s3_bucket_lifecycle_configuration"
    r.change.after.bucket == resource.change.after.bucket
  ]) > 0

  not lifecycle_exists
  msg := sprintf("S3 bucket %s has no lifecycle configuration — storage costs will grow unbounded", [resource.address])
}

# Warn on large storage classes being used for new buckets
warn[msg] {
  resource := input.resource_changes[_]
  resource.type == "aws_s3_bucket_lifecycle_configuration"
  not resource.change.after.rule[_].transition

  msg := sprintf("S3 bucket lifecycle in %s has no storage class transitions — consider moving infrequent data to STANDARD_IA or GLACIER", [resource.address])
}

NAT Gateway Checks

NAT gateways cost ~$32/month each, plus $0.045/GB of data processed. A common mistake is creating NAT gateways per AZ when one would suffice for non-production environments:

# policy/nat_gateway.rego
package terraform.networking

# Count NAT gateways being created
nat_gateway_count := count([r |
  r := input.resource_changes[_]
  r.type == "aws_nat_gateway"
  r.change.actions[_] == "create"
])

# In non-prod, one NAT gateway is usually enough
deny[msg] {
  nat_gateway_count > 1
  environment := input.variables.environment.value
  environment != "production"
  msg := sprintf("Creating %d NAT gateways in %s environment — use 1 for non-prod to save ~$%d/month",
    [nat_gateway_count, environment, (nat_gateway_count - 1) * 32])
}

RDS Instance Size Checks

# policy/rds_cost.rego
package terraform.rds

# Production-only instance classes (block in non-prod)
expensive_classes := {"db.r6g.4xlarge", "db.r6g.8xlarge", "db.r6g.16xlarge",
                      "db.r5.4xlarge", "db.r5.8xlarge", "db.r5.12xlarge"}

deny[msg] {
  resource := input.resource_changes[_]
  resource.type == "aws_db_instance"
  resource.change.actions[_] == "create"

  class := resource.change.after.instance_class
  expensive_classes[class]

  environment := input.variables.environment.value
  environment != "production"

  msg := sprintf("Instance class %s is not allowed in %s — use db.t3.medium or smaller",
    [class, environment])
}

# Multi-AZ should only be production
deny[msg] {
  resource := input.resource_changes[_]
  resource.type == "aws_db_instance"
  resource.change.after.multi_az == true

  environment := input.variables.environment.value
  environment != "production"

  msg := sprintf("Multi-AZ RDS is not allowed in %s environment — adds ~$%d/month",
    [environment, 50])
}

AWS Cost Anomaly Detection Integration

AWS Cost Anomaly Detection uses ML to identify unusual spending patterns. Integrate it with your infrastructure as code and alert on anomalies:

# terraform/cost-monitoring.tf
resource "aws_ce_anomaly_monitor" "main" {
  name         = "AnomalyMonitor"
  monitor_type = "DIMENSIONAL"

  monitor_dimension = "SERVICE"
}

resource "aws_ce_anomaly_subscription" "main" {
  name      = "AnomalySubscription"
  frequency = "DAILY"

  monitor_arn_list = [aws_ce_anomaly_monitor.main.arn]

  subscriber {
    address = var.cost_alert_email
    type    = "EMAIL"
  }

  # Alert when anomaly exceeds $50 or 20% above baseline
  threshold_expression {
    or {
      dimension {
        key           = "ANOMALY_TOTAL_IMPACT_ABSOLUTE"
        values        = ["50"]
        match_options = ["GREATER_THAN_OR_EQUAL"]
      }
      dimension {
        key           = "ANOMALY_TOTAL_IMPACT_PERCENTAGE"
        values        = ["20"]
        match_options = ["GREATER_THAN_OR_EQUAL"]
      }
    }
  }
}

# SNS integration for Slack/PagerDuty
resource "aws_sns_topic" "cost_alerts" {
  name = "cost-anomaly-alerts"
}

resource "aws_ce_anomaly_subscription" "sns" {
  name      = "AnomalySubscriptionSNS"
  frequency = "IMMEDIATE"  # immediate for severe anomalies

  monitor_arn_list = [aws_ce_anomaly_monitor.main.arn]

  subscriber {
    address = aws_sns_topic.cost_alerts.arn
    type    = "SNS"
  }

  threshold_expression {
    dimension {
      key           = "ANOMALY_TOTAL_IMPACT_ABSOLUTE"
      values        = ["500"]
      match_options = ["GREATER_THAN_OR_EQUAL"]
    }
  }
}

Lambda for Anomaly Alerting to Slack

// handlers/costAnomalyAlert.js
export const handler = async (event) => {
  const record = event.Records[0].Sns;
  const message = JSON.parse(record.Message);

  const anomaly = message.anomalyDetails;
  const slackMessage = {
    text: `:warning: *AWS Cost Anomaly Detected*`,
    blocks: [
      {
        type: "section",
        text: {
          type: "mrkdwn",
          text: [
            `*Service:* ${anomaly.dimensionValue}`,
            `*Impact:* $${anomaly.impact.totalImpact.toFixed(2)} (${anomaly.impact.totalImpactPercentage.toFixed(0)}% above normal)`,
            `*Period:* ${anomaly.anomalyStartDate} to ${anomaly.anomalyEndDate}`,
            `*Root causes:* ${anomaly.rootCauses.map(r => r.service).join(", ")}`,
          ].join("\n"),
        },
      },
      {
        type: "actions",
        elements: [{
          type: "button",
          text: { type: "plain_text", text: "View in AWS Console" },
          url: `https://console.aws.amazon.com/cost-management/home#/anomaly-detection/overview`,
        }],
      },
    ],
  };

  await fetch(process.env.SLACK_WEBHOOK_URL, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify(slackMessage),
  });
};

Policy-as-Code for Cost Governance with OPA

For organization-wide cost governance, define policies that run against every Terraform plan across all repos:

# policies/cost_governance.rego
package cost_governance

import future.keywords.in

# Resource cost limits by environment
max_monthly_cost := {
  "development": 50,
  "staging":     200,
  "production":  10000,
}

# Deny resources that exceed per-resource cost limits
deny[msg] {
  resource := input.resource_changes[_]
  resource.change.actions[_] == "create"

  # Get estimated cost from Infracost annotations (if available)
  estimated_cost := resource.metadata.infracostMonthlyCost
  environment := input.variables.environment.value

  limit := max_monthly_cost[environment]
  estimated_cost > limit

  msg := sprintf(
    "%s would cost $%.2f/month in %s environment (limit: $%d)",
    [resource.address, estimated_cost, environment, limit]
  )
}

# Require cost center tags on all resources
deny[msg] {
  resource := input.resource_changes[_]
  resource.change.actions[_] == "create"
  taggable_resources := {
    "aws_instance", "aws_db_instance", "aws_elasticache_cluster",
    "aws_eks_cluster", "aws_s3_bucket"
  }
  taggable_resources[resource.type]
  not resource.change.after.tags["CostCenter"]
  msg := sprintf("%s is missing required CostCenter tag", [resource.address])
}

# Require approval for changes over $100/month
warn[msg] {
  total_increase := sum([c |
    r := input.resource_changes[_]
    r.change.actions[_] == "create"
    c := r.metadata.infracostMonthlyCost
  ])
  total_increase > 100
  msg := sprintf(
    "This change increases monthly costs by $%.2f — requires manager approval",
    [total_increase]
  )
}

Integrate with a conftest policy server for centralized governance:

# .github/workflows/terraform.yml
- name: OPA cost policy check
  run: |
    terraform show -json tfplan > plan.json
    
    # Enrich plan with Infracost data
    infracost breakdown --path=. --format=json > infracost.json
    
    # Merge Infracost costs into plan
    jq -s '.[0] * {"infracost": .[1]}' plan.json infracost.json > enriched-plan.json
    
    # Run OPA policies
    opa eval \
      --data policies/ \
      --input enriched-plan.json \
      --format pretty \
      "data.cost_governance.deny" | tee policy-violations.txt
    
    VIOLATIONS=$(opa eval \
      --data policies/ \
      --input enriched-plan.json \
      "data.cost_governance.deny" | jq '.result[0].expressions[0].value | length')
    
    if [ "$VIOLATIONS" -gt 0 ]; then
      echo "Cost policy violations found:"
      cat policy-violations.txt
      exit 1
    fi

Scheduled Cost Regression Tests

Beyond PR-time checks, run scheduled tests to detect cost drift in existing infrastructure:

#!/bin/bash
# scripts/cost-regression-check.sh
# Run daily via cron or GitHub Actions scheduled workflow

# Get current month's costs by service
aws ce get-cost-and-usage \
  --time-period Start=$(date -d '7 days ago' +%Y-%m-%d),End=$(date +%Y-%m-%d) \
  --granularity DAILY \
  --metrics BlendedCost \
  --group-by Type=DIMENSION,Key=SERVICE \
  --output json > current-costs.json

# Compare against baseline (stored in S3)
aws s3 cp s3://my-cost-baselines/baseline.json baseline.json

# Check for services that increased more than 50% week-over-week
node -e "
const current = require('./current-costs.json');
const baseline = require('./baseline.json');

const currentByService = {};
current.ResultsByTime.forEach(day => {
  day.Groups.forEach(g => {
    const service = g.Keys[0];
    const cost = parseFloat(g.Metrics.BlendedCost.Amount);
    currentByService[service] = (currentByService[service] || 0) + cost;
  });
});

const baselineByService = {};
baseline.ResultsByTime.forEach(day => {
  day.Groups.forEach(g => {
    const service = g.Keys[0];
    const cost = parseFloat(g.Metrics.BlendedCost.Amount);
    baselineByService[service] = (baselineByService[service] || 0) + cost;
  });
});

let hasAnomaly = false;
Object.entries(currentByService).forEach(([service, cost]) => {
  const base = baselineByService[service] || 0;
  if (base > 1 && cost > base * 1.5) {
    console.error(\`ANOMALY: \${service} increased from \$\${base.toFixed(2)} to \$\${cost.toFixed(2)} (+\${((cost/base - 1) * 100).toFixed(0)}%)\`);
    hasAnomaly = true;
  }
});

if (hasAnomaly) process.exit(1);
console.log('No cost anomalies detected');
"

Schedule it in GitHub Actions:

name: Cost Regression Check
on:
  schedule:
    - cron: "0 9 * * 1-5"  # 9 AM UTC, Monday to Friday

jobs:
  cost-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.COST_EXPLORER_ROLE }}
          aws-region: us-east-1
      - run: bash scripts/cost-regression-check.sh

Conclusion

Cloud cost testing is infrastructure testing applied to the financial dimension of your system. Infracost brings cost visibility to PR reviews, converting abstract resource changes into dollar figures that engineers and managers both understand. OPA policies encode cost governance rules that run automatically, preventing expensive configurations from ever reaching production. AWS Cost Anomaly Detection catches the runtime surprises that static analysis misses.

The goal is not to block engineers from using the resources they need — it's to make cost implications explicit, automatic, and visible at the earliest possible moment. A $2,000/month RDS upgrade mentioned in a PR comment is a conversation; the same change discovered on the monthly AWS bill is a post-mortem.