Security Regression Testing: Keeping Compliance Continuous

Security regression testing is the practice of continuously verifying that security controls remain effective as codebases evolve. It's the discipline that prevents a passing security audit in January from becoming a failed one in December. Without it, every feature deployment is a potential compliance event — and you won't know until an auditor or attacker tells you.

This guide covers what security regression testing means in practice, how to integrate static and dynamic analysis tools into your CI pipeline, how to track security test coverage, and how to build alerting systems that catch regressions before they become findings.

What Security Regression Testing Actually Means

"Security regression" is when a system that previously met a security requirement no longer does. This happens in several ways:

Code changes that re-introduce vulnerabilities: A developer refactors an input handling function and removes a sanitization step. A SQL injection that was fixed three years ago comes back in a new module that replicates the old pattern.

Dependency updates that introduce new CVEs: A library update ships with a new vulnerability. The code didn't change, but the risk profile did.

Configuration drift: An infrastructure change removes a security group rule, weakens a TLS configuration, or enables a debug endpoint in production. No code changed, but controls degraded.

Permission creep: Access control rules that were tight gradually loosen as exceptions accumulate. Over months, a role that should have read-only access to a subset of data ends up with write access to everything.

Architectural changes that expand attack surface: A new microservice is added to the cardholder data environment without going through the same security review as existing services.

Security regression testing is the systematic detection of all of these. It requires multiple layers: SAST for code-level regressions, DAST for behavior-level regressions, configuration scanning for infrastructure regressions, and dependency scanning for supply chain regressions.

SAST in CI: Catching Code-Level Regressions

Static Application Security Testing (SAST) analyzes source code without executing it. For security regression testing, SAST is most useful when it runs on every pull request and is configured to fail on findings that meet or exceed a defined severity threshold.

Configuration Principles

Don't just run SAST — enforce it. A SAST scan that produces findings but doesn't block the build is a decoration. For it to function as a regression gate, a finding above your severity threshold must prevent merge.

Start with a baseline. On legacy codebases, starting with zero tolerance is impractical — there will be hundreds of existing findings. Instead, take an initial scan, acknowledge existing findings as accepted risks (with documented rationale), and configure the tool to fail only on new findings. This is the "ratchet" approach.

Configure for your tech stack. Generic SAST rules produce high false-positive rates. Use rules configured for your actual languages and frameworks. Semgrep's community registry has framework-specific rule sets for Django, Flask, Express, Spring, Rails, and many others.

Semgrep Configuration Example

# .semgrep.yml — placed in repo root
rules:
  - id: sql-injection-format-string
    patterns:
      - pattern: |
          db.execute("... %s ..." % ...)
    message: "SQL injection via string formatting"
    severity: ERROR
    languages: [python]
    metadata:
      pci_dss: "6.2.4"
      cwe: "CWE-89"

  - id: hardcoded-secret
    patterns:
      - pattern: |
          $KEY = "..."
      - metavariable-regex:
          metavariable: $KEY
          regex: (password|secret|api_key|token|private_key)
    message: "Potential hardcoded secret"
    severity: ERROR
    languages: [python, javascript, java]
    metadata:
      pci_dss: "8.6"

# .github/workflows/sast.yml
name: SAST Security Gate

on: [pull_request]

jobs:
  semgrep:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Semgrep scan
        uses: returntocorp/semgrep-action@v1
        with:
          config: >-
            p/owasp-top-ten
            p/sql-injection
            p/secrets
            .semgrep.yml
          # Fail on any ERROR severity finding
          # auditFailureThreshold: error
        env:
          SEMGREP_APP_TOKEN: ${{ secrets.SEMGREP_APP_TOKEN }}

      - name: Upload SARIF results
        uses: github/codeql-action/upload-sarif@v2
        if: always()
        with:
          sarif_file: semgrep.sarif

Tracking Findings Over Time

A SAST tool that runs on every PR but whose findings are never tracked is producing data that evaporates. For security regression testing to work, you need a findings lifecycle:

1. New finding detected → auto-create ticket with severity and control mapping
2. Finding triaged → accepted risk (documented) or assigned for remediation
3. Finding remediated → closed with remediation commit reference
4. Accepted risks reviewed → quarterly, accepted risks re-evaluated

Most commercial SAST tools have findings management built in. For open source tools, integrate with your issue tracker via the CI pipeline:

# scripts/sast_findings_to_jira.py
import json, requests

def process_sarif(sarif_file, jira_project):
    with open(sarif_file) as f:
        sarif = json.load(f)
    
    for run in sarif['runs']:
        for result in run['results']:
            if result['level'] in ('error', 'warning'):
                create_or_update_jira_issue(
                    project=jira_project,
                    summary=f"[SAST] {result['message']['text'][:100]}",
                    description=format_finding(result),
                    labels=['security', 'sast', result['level']],
                    # Deduplicate by rule + file + line
                    dedup_key=f"{result['ruleId']}:{result['locations'][0]['physicalLocation']['artifactLocation']['uri']}:{result['locations'][0]['physicalLocation']['region']['startLine']}"
                )

DAST in CI: Catching Behavior-Level Regressions

Dynamic Application Security Testing (DAST) tests the running application. It catches vulnerabilities that don't appear in static analysis: authentication flaws, session management issues, business logic vulnerabilities, and runtime misconfigurations.

DAST in CI requires a deployed, running application — typically your staging environment. It runs later in the pipeline than SAST and takes longer to execute.

OWASP ZAP Integration

OWASP ZAP is the standard open source DAST tool. For CI integration, the ZAP baseline scan is the right starting point — it runs a set of passive checks against your application without brute-forcing or actively exploiting vulnerabilities.

# GitHub Actions — DAST on deployment to staging
dast-scan:
  needs: deploy-staging
  runs-on: ubuntu-latest
  steps:
    - name: ZAP Baseline Scan
      uses: zaproxy/action-baseline@v0.9.0
      with:
        target: 'https://staging.yourapp.com'
        rules_file_name: '.zap/rules.tsv'
        cmd_options: '-a'  # Include alpha rules
        fail_action: true  # Fail on WARN or above
        artifact_name: 'zap-report'

    - name: ZAP Full Scan (scheduled only)
      if: github.event_name == 'schedule'
      uses: zaproxy/action-full-scan@v0.8.0
      with:
        target: 'https://staging.yourapp.com'
        rules_file_name: '.zap/rules.tsv'

Configuring rules for your application: ZAP will flag issues that aren't vulnerabilities in your context — false positives that create noise. Suppress them explicitly:

# .zap/rules.tsv — suppress specific rules
# Format: rule_id  IGNORE/WARN/FAIL  optional_reason
10202  IGNORE  # Password Autocomplete - by design
40012  IGNORE  # XSS reflected - handled by CSP
10021  FAIL    # X-Content-Type-Options - must be set
10038  FAIL    # Content Security Policy - must be set

Authenticated DAST Scans

Many critical vulnerabilities only appear behind authentication. An unauthenticated DAST scan misses most of your application surface. Configure ZAP with session scripts or context files to authenticate:

# zap_auth_script.py — ZAP authentication script
import urllib.request

def authenticate(helper, paramsValues, credentials):
    """Called by ZAP to authenticate the session"""
    login_url = paramsValues.get('Login URL')
    
    data = urllib.parse.urlencode({
        'email': credentials.getParam('Username'),
        'password': credentials.getParam('Password')
    }).encode()
    
    request = urllib.request.Request(login_url, data)
    response = urllib.request.urlopen(request)
    
    # Return session token from response
    return response.read().decode()

def getRequiredParamsNames():
    return ['Login URL']

def getCredentialsParamsNames():
    return ['Username', 'Password']

Tracking Security Test Coverage

Security test coverage answers: "What percentage of our security requirements have corresponding automated tests?" It's a different question from code coverage, and it requires different measurement.

Defining Your Security Test Coverage Baseline

Start with your compliance requirements. For PCI-DSS, list every requirement that has a technical control (not just a policy). For SOC 2, list every control in scope. Mark each one:

• Covered by automated test: Specific test exists, runs in CI, pass/fail tracked
• Covered by automated scan: Tool runs against this control class regularly
• Covered by manual test: Run periodically by a human, results recorded
• Not covered: No test exists

A simple spreadsheet works. The goal is visibility, not tooling sophistication.

| Requirement | Description | Coverage Type | Test/Tool | Last Run |
|---|---|---|---|---|
| PCI 8.3.1 | Min 12-char passwords | Automated test | `TestPasswordPolicy::test_minimum_length` | Every PR |
| PCI 8.4.2 | MFA for remote access | Automated test | `TestMFA::test_admin_requires_mfa` | Every PR |
| PCI 10.2.1 | Log all user access | Automated test | `TestAuditLogging::test_event_captured` | Every PR |
| PCI 11.3.1 | Quarterly vuln scan | Automated scan | Trivy + Prowler | Weekly |
| PCI 11.4.3 | Annual pen test | Manual | External vendor | Annually |
| PCI 2.2.1 | System hardening | Automated scan | OpenSCAP | On image build |

Automate the generation of this table by parsing your test suite's markers and your CI scan configurations. Any requirement row with no automated coverage is a gap to address.

Coverage Regression Alerting

When someone deletes a compliance test — intentionally or not — that's a coverage regression. Detect it:

# scripts/check_coverage_regression.py
import subprocess, json

def get_current_compliance_tests():
    """Parse test suite for all @pytest.mark.compliance tests"""
    result = subprocess.run([
        'pytest', 'tests/compliance/', '--collect-only', '-q', '--no-header'
    ], capture_output=True, text=True)
    return set(result.stdout.splitlines())

def check_regression(baseline_file):
    with open(baseline_file) as f:
        baseline = set(json.load(f))
    
    current = get_current_compliance_tests()
    removed = baseline - current
    
    if removed:
        print(f"COVERAGE REGRESSION: {len(removed)} compliance tests removed:")
        for test in sorted(removed):
            print(f"  - {test}")
        return 1
    return 0

Run this in CI and fail the build if the compliance test count drops below the baseline.

Preventing Compliance Drift

Compliance drift — the gradual degradation of controls between audit cycles — is the most common cause of audit surprises. Preventing it requires monitoring the controls themselves, not just the code.

Infrastructure Configuration Monitoring

Use AWS Config, Azure Policy, or GCP Security Command Center to continuously monitor cloud resources against your compliance baseline. When a resource drifts, create a ticket automatically:

# Lambda function triggered by AWS Config rule violation
import boto3, json

def handle_config_violation(event, context):
    violation = event['detail']
    resource_type = violation['resourceType']
    resource_id = violation['resourceId']
    rule_name = violation['configRuleName']
    
    # Create ticket in your issue tracker
    create_ticket(
        title=f"Compliance drift: {rule_name} on {resource_type}/{resource_id}",
        priority='high',
        labels=['compliance', 'drift', 'auto-detected'],
        body=f"""
        AWS Config detected a compliance violation.
        
        Rule: {rule_name}
        Resource: {resource_type}/{resource_id}
        Time: {violation['notificationCreationTime']}
        
        This must be remediated within SLA or formally accepted as a risk.
        """
    )

Access Review Automation

Manual quarterly access reviews fail because they depend on someone remembering to do them. Automate the reminder, the data collection, and the record-keeping:

#!/bin/bash
# scripts/quarterly_access_review.sh — run as scheduled CI job

# Pull current access lists from identity provider
okta-cli users list --format json > /tmp/current-access.json

# Compare against previous review
python3 scripts/diff_access.py \
  --previous compliance-evidence/access-reviews/last-review.json \
  --current /tmp/current-access.json \
  --output compliance-evidence/access-reviews/$(date +%Y-%m-%d)-access-review.json

# Generate review document for human sign-off
python3 scripts/generate_access_review.py \
  compliance-evidence/access-reviews/$(date +%Y-%m-%d)-access-review.json \
  > compliance-evidence/access-reviews/$(date +%Y-%m-%d)-review-doc.md

# Notify review owner
gh issue create \
  --title "Quarterly Access Review — $(date +%Y-Q%q)" \
  --body "$(cat compliance-evidence/access-reviews/$(date +%Y-%m-%d)-review-doc.md)" \
  --label "compliance,access-review" \
  --assignee @security-team

Alerting on Security Regressions

Alerting is the runtime layer of security regression testing. Your tests catch regressions before deployment; alerting catches them in production.

What to Alert On

Authentication anomalies:

• Login failure rate > threshold (brute force indicator)
• Login from new geolocation for privileged user
• MFA bypass attempts
• Session token reuse after logout

Authorization failures:

• Spike in 403 responses from a single user (probe indicator)
• Access to admin endpoints from non-admin role
• Unusual data volume accessed by a single session

Audit log anomalies:

• Gap in audit log sequence (deletion indicator)
• Audit log write failures
• Unusual volume of high-privilege events

Infrastructure drift:

• Security group rule added or modified
• IAM policy attached to user directly (vs. via role)
• Public access enabled on storage resource
• TLS certificate approaching expiration

Alert Routing by Severity

Not all alerts have the same urgency. Define severity levels and corresponding response SLAs:

Document these SLAs. SOC 2 CC7.4 requires evidence that security events are evaluated and responded to in a timely manner. Your PagerDuty acknowledge timestamps and JIRA close timestamps are that evidence.

Putting It Together: The Security Regression Pipeline

The complete security regression testing pipeline looks like this:

Every commit/PR:
  SAST scan (Semgrep/CodeQL) → block on new HIGH/CRITICAL
  Secret detection (truffleHog/gitleaks) → block on any finding
  Dependency scan (Snyk/Trivy) → block on CRITICAL

Every deployment to staging:
  DAST scan (ZAP baseline) → block on WARN+
  Compliance test suite → block on any failure
  Coverage regression check → block if compliance tests deleted

Weekly scheduled:
  Cloud posture scan (Prowler) → ticket on any FAIL
  Container image full scan → ticket on new HIGH+
  OpenSCAP host scan → ticket if score drops

Monthly:
  Access review generation → ticket requiring human sign-off
  Vendor SOC 2 report review → ticket if reports expired

Runtime / continuous:
  SIEM alerting on authentication anomalies → PagerDuty
  AWS Config drift detection → ticket
  Certificate expiry monitoring → ticket at 30/14/7 days

The key principle: every layer catches different failure modes. SAST catches code-level regressions that DAST misses because the vulnerability never fires in testing. DAST catches runtime configuration issues that SAST can't see. Cloud posture scanning catches infrastructure drift that neither SAST nor DAST tests. Runtime alerting catches active exploitation attempts that all the pre-production testing missed.

Security regression testing is engineering discipline applied to compliance. The teams that do it well treat each compliance control as a system requirement, write tests for it, and monitor it continuously. They're not surprised at audit time because they already know what their controls look like. They're not scrambling for evidence because they've been generating it for twelve months. And they're not failing penetration tests because they ran their own regression suite last week.