Testing

Continuous Infrastructure Validation in CI/CD Pipelines

HelpMeTest

28 Aug 2026 — 8 min read

Infrastructure validation should happen at every stage of the development lifecycle, not just at apply time. By combining static analysis (terraform validate, tflint), security scanning (checkov), cost estimation (infracost), and smoke tests into a single CI pipeline, you catch problems when they're cheapest to fix—during code review, not after a production incident.

Key Takeaways

Layer validation from fast to slow. Run syntax checks first (seconds), then static analysis (tens of seconds), then security/cost scans (minutes), then integration tests (minutes to hours). Fail fast on cheap checks.

Make cost visible on every PR. Infracost posts a cost diff comment on every PR that changes infrastructure—this changes the conversation from "did it work?" to "did it work within budget?".

Checkov catches what humans miss. Security misconfigurations are systematic. Checkov knows 1000+ rules; your reviewer knows a few. Run both.

Smoke tests verify the actual deployed state. After apply, curl the endpoints, query the database, list the S3 objects—don't trust that Terraform success means the workload works.

Run infrastructure validation on every PR, not just on main. Shift-left means catching violations when the author is still in context and can fix them in minutes, not days.

The Infrastructure Validation Gap

Most teams validate infrastructure too late. The pattern is: engineer writes Terraform, creates a PR, reviewer approves based on code review, changes merge, Terraform applies, something breaks. The feedback loop is hours to days.

A proper infrastructure CI pipeline collapses this to minutes. By the time a PR is ready for human review, it should have already passed syntax checks, linting, security scanning, and cost estimation. Human reviewers should be making judgment calls, not spotting missing encryption settings.

Stage 1: Syntax and Format

The fastest checks are format and syntax validation. Fail immediately if these don't pass—no point running expensive checks on malformatted code.

# First stage: format and syntax
- name: Terraform Format Check
  run: terraform fmt -check -recursive
  # Fails if any .tf file is not formatted

- name: Terraform Validate
  run: |
    terraform init -backend=false
    terraform validate
  # Catches: missing required variables, invalid references, type errors

terraform validate requires init but you can use -backend=false to skip remote state configuration—this works without cloud credentials.

For HCL that uses modules, mock providers in your CI environment:

# tests/mocks/mock_providers.tf
# Prevents init from downloading actual providers
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region                      = "us-east-1"
  skip_credentials_validation = true
  skip_requesting_account_id  = true
  access_key                  = "mock_access_key"
  secret_key                  = "mock_secret_key"
}

Stage 2: Linting with tflint

tflint catches issues that terraform validate misses: deprecated arguments, invalid AMI references, incorrect instance types, unused variables.

Install and configure:

curl -Lo /tmp/tflint.zip https://github.com/terraform-linters/tflint/releases/latest/download/tflint_linux_amd64.zip
unzip /tmp/tflint.zip -d /usr/local/bin

Create .tflint.hcl in your repository root:

# .tflint.hcl
plugin "aws" {
  enabled = true
  version = "0.29.0"
  source  = "github.com/terraform-linters/tflint-ruleset-aws"
}

rule "terraform_documented_variables" {
  enabled = true
}

rule "terraform_documented_outputs" {
  enabled = true
}

rule "terraform_naming_convention" {
  enabled = true
  format  = "snake_case"
}

rule "terraform_unused_declarations" {
  enabled = true
}

rule "terraform_required_version" {
  enabled = true
}

Run:

tflint --init
tflint --recursive

tflint with the AWS plugin catches real problems like:

Using t2.micro when t3.micro is the current generation
Referencing availability zones that don't exist in the region
Invalid IAM policy JSON (via the policy validator plugin)

Stage 3: Security Scanning with Checkov

Checkov scans Terraform, CloudFormation, Kubernetes, Dockerfile, and more against 1000+ security rules from CIS Benchmarks, NIST, SOC2, and GDPR.

pip install checkov
checkov -d . --framework terraform --output cli --output junitxml --output-file-path results/

Critical checks Checkov catches out of the box:

S3 buckets without versioning or encryption
Security groups with 0.0.0.0/0 ingress on sensitive ports
RDS instances without deletion protection
Lambda functions without dead letter queues
IAM policies with wildcard actions ("Action": "*")
CloudTrail logging disabled
KMS keys without rotation

Configure which checks to skip (for intentional exceptions):

# .checkov.yml
skip-check:
  - CKV_AWS_144  # S3 cross-region replication (we don't need it)
  - CKV_AWS_18   # S3 access logging (handled at account level)
soft-fail: false
compact: true

For baseline comparisons—when you want to prevent new violations but not block on existing ones:

# Create baseline from current state
checkov -d . --create-baseline

# Future runs only fail on new violations
checkov -d . --baseline .checkov.baseline

Stage 4: Cost Estimation with Infracost

Infracost runs terraform plan and estimates the monthly cost of the planned infrastructure. The killer feature: it posts a cost diff comment on every PR.

brew install infracost
infracost auth login

Generate a cost estimate:

infracost breakdown --path . --format json > infracost.json
infracost output --path infracost.json --format table

Output:

Project: my-infrastructure

 Name                                    Monthly Qty  Unit   Monthly Cost
 ─────────────────────────────────────────────────────────────────────────
 aws_instance.api_server
 ├─ Instance usage (Linux/UNIX, on-demand, m5.xlarge)   730  hours        $140.16
 └─ root_block_device
    └─ Storage (general purpose SSD, gp3)                50  GB             $4.00

 aws_db_instance.postgres
 ├─ Database instance (on-demand, db.t3.medium)         730  hours         $52.56
 └─ Storage (general purpose SSD, gp2)                 100  GB             $11.50

 OVERALL TOTAL                                                            $208.22

In CI, generate a diff against the base branch:

# On the PR branch
infracost breakdown --path . --format json > infracost-new.json

# Diff against the base (requires fetching the base plan)
git checkout main
infracost breakdown --path . --format json > infracost-base.json
git checkout -

infracost diff \
  --path infracost-new.json \
  --compare-to infracost-base.json \
  --format github-comment > comment.md

Stage 5: Plan Review

Before applying, generate and store the plan file for review:

terraform plan -out=tfplan.binary
terraform show -json tfplan.binary > tfplan.json

Use tfplan.json for:

Automated checks (custom scripts that parse the plan)
Human review artifacts (store in S3, link from PR)
Input to conftest for policy checks

# Run conftest against the plan output
terraform show -json tfplan.binary | conftest test - --policy policy/

This catches policy violations against the actual planned changes, not just the source code.

Stage 6: Smoke Tests After Apply

A successful terraform apply means Terraform didn't error. It does not mean your workload is healthy. Always run smoke tests after apply:

#!/bin/bash
# scripts/smoke-test.sh
set -euo pipefail

API_URL=$(terraform output -raw api_endpoint)
DB_HOST=$(terraform output -raw db_host)
BUCKET=$(terraform output -raw assets_bucket)

echo "=== Smoke Tests ==="

# Test 1: API responds
echo -n "API health check... "
HTTP_STATUS=$(curl -sf -o /dev/null -w "%{http_code}" "${API_URL}/health")
if [ "$HTTP_STATUS" != "200" ]; then
    echo "FAIL (HTTP $HTTP_STATUS)"
    exit 1
fi
echo "OK"

# Test 2: Database connectivity (via API)
echo -n "Database connectivity... "
HTTP_STATUS=$(curl -sf -o /dev/null -w "%{http_code}" "${API_URL}/health/db")
if [ "$HTTP_STATUS" != "200" ]; then
    echo "FAIL (HTTP $HTTP_STATUS)"
    exit 1
fi
echo "OK"

# Test 3: S3 bucket exists and is accessible
echo -n "S3 bucket accessible... "
aws s3 ls "s3://${BUCKET}" > /dev/null 2>&1
echo "OK"

# Test 4: TLS certificate is valid
echo -n "TLS certificate valid... "
DOMAIN=$(echo "$API_URL" | sed 's|https://||' | sed 's|/.*||')
CERT_EXPIRY=$(echo | openssl s_client -servername "$DOMAIN" -connect "${DOMAIN}:443" 2>/dev/null \
    | openssl x509 -noout -dates 2>/dev/null | grep notAfter | cut -d= -f2)
echo "Expires: $CERT_EXPIRY OK"

echo "=== All smoke tests passed ==="

Complete GitHub Actions Pipeline

# .github/workflows/terraform-ci.yml
name: Infrastructure CI

on:
  pull_request:
    paths:
      - '**.tf'
      - '**.tfvars'
  push:
    branches: [main]

env:
  TF_VERSION: "1.7.0"
  AWS_DEFAULT_REGION: us-east-1

jobs:
  fmt-validate:
    name: Format & Validate
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: ${{ env.TF_VERSION }}
      - name: Format check
        run: terraform fmt -check -recursive
      - name: Validate
        run: |
          terraform init -backend=false
          terraform validate

  tflint:
    name: TFLint
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: terraform-linters/setup-tflint@v4
        with:
          tflint_version: v0.50.0
      - name: Init TFLint
        run: tflint --init
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      - name: Run TFLint
        run: tflint --recursive --format compact

  security-scan:
    name: Checkov Security Scan
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run Checkov
        uses: bridgecrewio/checkov-action@v12
        with:
          directory: .
          framework: terraform
          output_format: cli,sarif
          output_file_path: console,results.sarif
          soft_fail: false
      - name: Upload SARIF
        uses: github/codeql-action/upload-sarif@v3
        if: always()
        with:
          sarif_file: results.sarif

  cost-estimate:
    name: Cost Estimate
    runs-on: ubuntu-latest
    if: github.event_name == 'pull_request'
    steps:
      - uses: actions/checkout@v4
      - name: Setup Infracost
        uses: infracost/actions/setup@v3
        with:
          api-key: ${{ secrets.INFRACOST_API_KEY }}
      - name: Generate cost estimate for PR branch
        run: infracost breakdown --path . --format json > /tmp/infracost-pr.json
      - name: Generate cost estimate for base branch
        run: |
          git fetch origin ${{ github.base_ref }}
          git checkout origin/${{ github.base_ref }} -- .
          infracost breakdown --path . --format json > /tmp/infracost-base.json
          git checkout HEAD -- .
      - name: Post PR comment
        run: |
          infracost diff \
            --path /tmp/infracost-pr.json \
            --compare-to /tmp/infracost-base.json \
            --format github-comment \
            --show-skipped | \
            infracost comment github \
              --path /tmp/infracost-pr.json \
              --repo ${{ github.repository }} \
              --pull-request ${{ github.event.pull_request.number }} \
              --github-token ${{ secrets.GITHUB_TOKEN }} \
              --behavior update

  plan:
    name: Terraform Plan
    runs-on: ubuntu-latest
    needs: [fmt-validate, tflint, security-scan]
    if: github.event_name == 'pull_request'
    environment: plan
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: ${{ env.TF_VERSION }}
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_PLAN_ROLE_ARN }}
          aws-region: ${{ env.AWS_DEFAULT_REGION }}
      - name: Terraform Init
        run: terraform init
      - name: Terraform Plan
        run: terraform plan -out=tfplan.binary
      - name: Convert plan to JSON
        run: terraform show -json tfplan.binary > tfplan.json
      - name: Policy check on plan
        run: |
          pip install conftest
          cat tfplan.json | conftest test - --policy policy/
      - name: Upload plan
        uses: actions/upload-artifact@v4
        with:
          name: tfplan
          path: |
            tfplan.binary
            tfplan.json

  apply:
    name: Terraform Apply
    runs-on: ubuntu-latest
    needs: [plan]
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    environment: production
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: ${{ env.TF_VERSION }}
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_APPLY_ROLE_ARN }}
          aws-region: ${{ env.AWS_DEFAULT_REGION }}
      - name: Download plan
        uses: actions/download-artifact@v4
        with:
          name: tfplan
      - name: Terraform Apply
        run: terraform apply -auto-approve tfplan.binary
      - name: Run smoke tests
        run: bash scripts/smoke-test.sh

  notify-failure:
    name: Notify on Failure
    runs-on: ubuntu-latest
    needs: [apply]
    if: failure()
    steps:
      - name: Send alert
        run: |
          curl -X POST "${{ secrets.SLACK_WEBHOOK }}" \
            -H "Content-Type: application/json" \
            -d '{"text": "Infrastructure deployment failed on main. Check the Actions run."}'

GitLab CI Equivalent

# .gitlab-ci.yml
stages:
  - validate
  - analyze
  - plan
  - apply
  - verify

variables:
  TF_VERSION: "1.7.0"
  AWS_DEFAULT_REGION: us-east-1

.terraform_base:
  image: hashicorp/terraform:${TF_VERSION}
  before_script:
    - terraform init

fmt-check:
  stage: validate
  image: hashicorp/terraform:${TF_VERSION}
  script:
    - terraform fmt -check -recursive

validate:
  stage: validate
  extends: .terraform_base
  script:
    - terraform validate

tflint:
  stage: analyze
  image: ghcr.io/terraform-linters/tflint:v0.50.0
  script:
    - tflint --init
    - tflint --recursive

checkov:
  stage: analyze
  image: bridgecrew/checkov:latest
  script:
    - checkov -d . --framework terraform --output cli
  allow_failure: false

infracost:
  stage: analyze
  image: infracost/infracost:ci-0.10
  only:
    - merge_requests
  script:
    - infracost breakdown --path . --format json > /tmp/infracost.json
    - infracost output --path /tmp/infracost.json --format table

plan:
  stage: plan
  extends: .terraform_base
  only:
    - merge_requests
  script:
    - terraform plan -out=tfplan.binary
    - terraform show -json tfplan.binary > tfplan.json
  artifacts:
    paths:
      - tfplan.binary
      - tfplan.json
    expire_in: 1 week

apply:
  stage: apply
  extends: .terraform_base
  only:
    - main
  when: manual
  script:
    - terraform apply -auto-approve
  environment:
    name: production

smoke-test:
  stage: verify
  only:
    - main
  needs: [apply]
  script:
    - bash scripts/smoke-test.sh

Measuring Pipeline Effectiveness

Track these metrics to understand if your pipeline is actually helping:

Time to feedback: How long from PR open to first CI result? Target under 10 minutes for the fast checks.
False positive rate: How often do CI failures block work that was actually correct? High false positives erode trust.
Escape rate: How often do violations make it past CI to production? Should trend toward zero.
Cost variance: Track actual vs. estimated costs monthly. Large gaps indicate your estimates or tagging are wrong.

Conclusion

A complete infrastructure validation pipeline runs in layers: fast checks first, expensive checks last, smoke tests after apply. The total investment—a few hundred lines of CI configuration and a handful of policy files—pays back immediately in prevented incidents and faster code review cycles. The tooling (Terraform, tflint, Checkov, Infracost, Conftest) is all open source and well-maintained. There's no reason to validate infrastructure only at apply time when you could be catching problems at PR open time.