Testing

SOC 2 Testing for SaaS: What Engineers Need to Verify

HelpMeTest

HelpMeTest

— 8 min read
SOC 2 Testing for SaaS: What Engineers Need to Verify

SOC 2 has become the de facto trust signal for B2B SaaS companies. Enterprise buyers require it; without it, deals stall in security review. The American Institute of CPAs (AICPA) defines SOC 2 around five Trust Service Criteria (TSC): Security, Availability, Processing Integrity, Confidentiality, and Privacy. Most companies pursue Security (CC series) plus one or two others relevant to their product.

What many engineering teams don't realize is that SOC 2 is fundamentally an evidence problem. The auditor examines whether your controls existed and operated effectively during a defined observation period (typically 6–12 months). Automated testing is how you generate consistent, datable evidence that controls worked.

Trust Service Criteria and What They Require

Criteria Abbreviation Core Question Commonly Tested Controls
Security CC Are systems protected against unauthorized access? Access control, encryption, vulnerability management, incident response
Availability A Do systems operate as committed? Uptime monitoring, RTO/RPO, capacity management
Processing Integrity PI Is processing complete, accurate, and timely? Data validation, error handling, reconciliation
Confidentiality C Is confidential data protected? Encryption, access restrictions, disposal
Privacy P Is personal information collected/used appropriately? Consent, data subject rights, retention

Most early-stage SaaS companies pursue SOC 2 Type I (point-in-time) first, then Type II (over an observation period). Type II is what enterprise buyers actually care about — it proves controls operated continuously, not just that they were in place on one day.

CC6: Logical and Physical Access Controls

CC6 is the largest control area in SOC 2 Security. It covers how you prevent unauthorized access to systems and data.

CC6.1 — Access Provisioning and Deprovisioning

# tests/soc2/test_access_controls.py
import pytest
import boto3
from datetime import datetime, timedelta
from myapp.api_client import APIClient

client = APIClient(base_url='https://api.example.com')
iam = boto3.client('iam')

def test_inactive_iam_users_do_not_exist():
    """
    SOC 2 CC6.1: Access is removed when no longer needed.
    IAM users with no activity in 90 days should be deprovisioned.
    """
    response = iam.generate_credential_report()
    # Wait for report to generate
    import time
    time.sleep(2)
    report = iam.get_credential_report()

    import csv
    import io
    reader = csv.DictReader(io.StringIO(report['Content'].read().decode('utf-8')))

    violations = []
    cutoff = datetime.utcnow() - timedelta(days=90)

    for user in reader:
        if user['user'] == '<root_account>':
            continue

        last_used = user.get('password_last_used', 'N/A')
        if last_used not in ('N/A', 'no_information', 'not_supported'):
            last_used_dt = datetime.strptime(last_used, '%Y-%m-%dT%H:%M:%S+00:00')
            if last_used_dt < cutoff:
                violations.append({
                    'user': user['user'],
                    'last_used': last_used,
                })

    assert len(violations) == 0, (
        f"Found {len(violations)} IAM users inactive for 90+ days: "
        + ', '.join(v['user'] for v in violations)
    )

def test_mfa_enabled_for_all_iam_users():
    """SOC 2 CC6.1: MFA required for all human IAM users."""
    users = iam.list_users()['Users']
    violations = []

    for user in users:
        mfa_devices = iam.list_mfa_devices(UserName=user['UserName'])['MFADevices']
        if len(mfa_devices) == 0:
            violations.append(user['UserName'])

    assert len(violations) == 0, (
        f"IAM users without MFA: {', '.join(violations)}"
    )

def test_root_account_has_no_access_keys():
    """SOC 2 CC6.1: Root account should not have programmatic access keys."""
    report = iam.get_credential_report()
    import csv, io
    reader = csv.DictReader(io.StringIO(report['Content'].read().decode('utf-8')))

    for user in reader:
        if user['user'] == '<root_account>':
            assert user['access_key_1_active'] == 'false', "Root account has active access key 1"
            assert user['access_key_2_active'] == 'false', "Root account has active access key 2"

def test_api_requires_authentication():
    """SOC 2 CC6.1: All protected endpoints require valid authentication."""
    protected_endpoints = [
        '/api/users',
        '/api/settings',
        '/api/billing',
        '/api/data-export',
    ]

    for endpoint in protected_endpoints:
        response = client.get(endpoint)
        assert response.status_code == 401, (
            f"Endpoint {endpoint} returned {response.status_code} without auth (expected 401)"
        )

CC6.3 — Role-Based Access Control

// tests/soc2/rbac.test.ts
import request from 'supertest';
import { app } from '../../src/app';

const ROLES = {
  admin: 'admin-token',
  member: 'member-token',
  viewer: 'viewer-token',
  billing_only: 'billing-token',
};

describe('SOC 2 CC6.3 — Role-Based Access Control', () => {
  const writeEndpoints = [
    { method: 'POST', path: '/api/users/invite' },
    { method: 'DELETE', path: '/api/users/some-user-id' },
    { method: 'PUT', path: '/api/settings/security' },
  ];

  writeEndpoints.forEach(({ method, path }) => {
    it(`${method} ${path} should be restricted to admins`, async () => {
      // Members and viewers should be forbidden
      for (const role of ['member', 'viewer', 'billing_only'] as const) {
        const res = await (request(app) as any)[method.toLowerCase()](path)
          .set('Authorization', `Bearer ${ROLES[role]}`);

        expect(res.status).toBe(403);
      }

      // Admins should be permitted (may still fail on payload, but not auth)
      const adminRes = await (request(app) as any)[method.toLowerCase()](path)
        .set('Authorization', `Bearer ${ROLES.admin}`);

      expect(adminRes.status).not.toBe(403);
    });
  });

  it('billing staff can only access billing endpoints', async () => {
    // Can access billing
    await request(app)
      .get('/api/billing/invoices')
      .set('Authorization', `Bearer ${ROLES.billing_only}`)
      .expect(200);

    // Cannot access user management
    await request(app)
      .get('/api/users')
      .set('Authorization', `Bearer ${ROLES.billing_only}`)
      .expect(403);
  });
});

CC7: System Operations — Logging and Monitoring

SOC 2 CC7 requires that you detect and respond to security events. This means logging, alerting, and verified detection capabilities.

// tests/soc2/logging.test.ts
import request from 'supertest';
import { app } from '../../src/app';
import { auditLog } from '../../src/audit-log';

describe('SOC 2 CC7 — Audit Logging', () => {
  it('successful login should generate an audit log entry', async () => {
    const spy = jest.spyOn(auditLog, 'write');

    await request(app)
      .post('/api/auth/login')
      .send({ email: 'user@example.com', password: 'ValidPass123!' })
      .expect(200);

    expect(spy).toHaveBeenCalledWith(
      expect.objectContaining({
        event_type: 'auth.login.success',
        user_email: 'user@example.com',
        timestamp: expect.any(String),
        ip_address: expect.any(String),
      })
    );
  });

  it('failed login should generate an audit log entry', async () => {
    const spy = jest.spyOn(auditLog, 'write');

    await request(app)
      .post('/api/auth/login')
      .send({ email: 'user@example.com', password: 'WrongPassword' })
      .expect(401);

    expect(spy).toHaveBeenCalledWith(
      expect.objectContaining({
        event_type: 'auth.login.failure',
        user_email: 'user@example.com',
      })
    );
  });

  it('privilege escalation should be logged', async () => {
    const spy = jest.spyOn(auditLog, 'write');

    await request(app)
      .post('/api/users/member-id/promote')
      .set('Authorization', 'Bearer admin-token')
      .send({ role: 'admin' })
      .expect(200);

    expect(spy).toHaveBeenCalledWith(
      expect.objectContaining({
        event_type: 'access.role.changed',
        target_user_id: 'member-id',
        new_role: 'admin',
        performed_by: expect.any(String),
      })
    );
  });

  it('audit logs should be immutable (no delete endpoint)', async () => {
    await request(app)
      .delete('/api/audit-logs/some-log-id')
      .set('Authorization', 'Bearer admin-token')
      .expect(404);
  });

  it('logs should be retained for minimum 12 months', async () => {
    // Test that old logs are not purged prematurely
    const { db } = await import('../../src/db');
    const oldLog = await db('audit_logs')
      .where('created_at', '<', db.raw("NOW() - INTERVAL '11 months'"))
      .first();

    // If we have data going back 11 months, verify it's still there
    if (oldLog) {
      expect(oldLog).toBeDefined();
    }
    // If not, we can't test this yet — but document the retention policy check
  });
});

CC8: Change Management

SOC 2 requires that changes to production systems go through a defined process — typically including code review, testing, and approval. These controls are partially enforced by your SCM and CI/CD pipeline.

# tests/soc2/test_change_management.py
import requests
import os

GITHUB_TOKEN = os.environ['GITHUB_TOKEN']
REPO = 'myorg/myapp'
HEADERS = {'Authorization': f'token {GITHUB_TOKEN}'}

def test_main_branch_requires_pull_request():
    """SOC 2 CC8.1: Changes to production branch require PR (no direct push)."""
    response = requests.get(
        f'https://api.github.com/repos/{REPO}/branches/main/protection',
        headers=HEADERS
    )
    assert response.status_code == 200, "Branch protection not configured on main"

    protection = response.json()
    assert protection.get('required_pull_request_reviews') is not None, \
        "Pull request reviews not required on main branch"

def test_pr_requires_minimum_reviewers():
    """SOC 2 CC8.1: PRs require at least one approving review."""
    response = requests.get(
        f'https://api.github.com/repos/{REPO}/branches/main/protection',
        headers=HEADERS
    )
    protection = response.json()
    pr_config = protection.get('required_pull_request_reviews', {})
    required_count = pr_config.get('required_approving_review_count', 0)

    assert required_count >= 1, (
        f"Expected at least 1 required reviewer, got {required_count}"
    )

def test_status_checks_required_before_merge():
    """SOC 2 CC8.1: CI tests must pass before merging to main."""
    response = requests.get(
        f'https://api.github.com/repos/{REPO}/branches/main/protection',
        headers=HEADERS
    )
    protection = response.json()
    checks = protection.get('required_status_checks', {})
    contexts = checks.get('contexts', [])

    required_checks = ['test', 'lint', 'security-scan']
    for check in required_checks:
        matching = [c for c in contexts if check in c.lower()]
        assert len(matching) > 0, (
            f"Required status check '{check}' not configured on main branch"
        )

def test_no_force_push_on_main():
    """SOC 2 CC8.1: Force pushes to production branch are blocked."""
    response = requests.get(
        f'https://api.github.com/repos/{REPO}/branches/main/protection',
        headers=HEADERS
    )
    protection = response.json()
    assert protection.get('allow_force_pushes', {}).get('enabled') == False, \
        "Force pushes allowed on main branch — this bypasses the change management process"

CC9 and A-Series: Availability Testing

If you've scoped Availability into your SOC 2, you need evidence that systems meet your committed uptime and that recovery procedures work.

# tests/soc2/test_availability.py
import requests
import time
import pytest

BASE_URL = 'https://api.example.com'

def test_health_endpoint_responds():
    """SOC 2 A1: Core system availability verified."""
    response = requests.get(f'{BASE_URL}/health', timeout=10)
    assert response.status_code == 200

    health = response.json()
    assert health.get('status') == 'healthy'
    assert health.get('database') == 'connected'
    assert health.get('cache') == 'connected'

def test_api_response_time_under_sla():
    """SOC 2 A1.2: System operates within committed performance levels."""
    endpoints = [
        '/api/users/me',
        '/api/projects',
        '/api/settings',
    ]

    SLA_MS = 2000  # 2 second SLA

    for endpoint in endpoints:
        start = time.time()
        response = requests.get(
            f'{BASE_URL}{endpoint}',
            headers={'Authorization': 'Bearer test-token'},
            timeout=10
        )
        elapsed_ms = (time.time() - start) * 1000

        assert elapsed_ms < SLA_MS, (
            f"{endpoint} took {elapsed_ms:.0f}ms, exceeding SLA of {SLA_MS}ms"
        )
        assert response.status_code in (200, 401), (
            f"{endpoint} returned unexpected status {response.status_code}"
        )

Automated Evidence Collection

SOC 2 audits require evidence. Instead of scrambling to collect screenshots and logs at audit time, automate evidence collection so it runs on schedule:

// scripts/soc2-evidence-collector.ts
import { writeFileSync, mkdirSync } from 'fs';
import { execSync } from 'child_process';
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';

const s3 = new S3Client({ region: 'us-east-1' });
const EVIDENCE_BUCKET = 'myapp-soc2-evidence';
const today = new Date().toISOString().split('T')[0];

async function collectEvidence() {
  const evidence: Record<string, unknown> = {
    collected_at: new Date().toISOString(),
    period: today,
  };

  // 1. Security group configurations
  const sgOutput = execSync(
    'aws ec2 describe-security-groups --filters Name=tag:Environment,Values=production --output json'
  ).toString();
  evidence.security_groups = JSON.parse(sgOutput);

  // 2. IAM credential report
  execSync('aws iam generate-credential-report');
  await new Promise(r => setTimeout(r, 3000));
  const credReport = execSync('aws iam get-credential-report --output json').toString();
  evidence.iam_credential_report = JSON.parse(credReport);

  // 3. CloudTrail status (audit logging active)
  const trailStatus = execSync(
    'aws cloudtrail get-trail-status --name prod-trail --output json'
  ).toString();
  evidence.cloudtrail_status = JSON.parse(trailStatus);

  // 4. Test results
  try {
    const testOutput = execSync('npx jest tests/soc2/ --json 2>/dev/null').toString();
    evidence.test_results = JSON.parse(testOutput);
  } catch (e: any) {
    evidence.test_results = { error: e.message };
  }

  const filename = `evidence-${today}.json`;
  const content = JSON.stringify(evidence, null, 2);

  // Save locally
  mkdirSync('soc2-evidence', { recursive: true });
  writeFileSync(`soc2-evidence/${filename}`, content);

  // Upload to S3 with retention policy
  await s3.send(new PutObjectCommand({
    Bucket: EVIDENCE_BUCKET,
    Key: `daily/${filename}`,
    Body: content,
    ContentType: 'application/json',
    ServerSideEncryption: 'aws:kms',
  }));

  console.log(`Evidence collected and uploaded: ${filename}`);
}

collectEvidence().catch(console.error);

Schedule this to run daily during your observation period:

# .github/workflows/soc2-evidence.yml
name: SOC 2 Evidence Collection

on:
  schedule:
    - cron: '0 8 * * *'  # Daily at 8am UTC

jobs:
  collect:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789:role/soc2-evidence-collector
          aws-region: us-east-1
      - name: Collect evidence
        run: npx ts-node scripts/soc2-evidence-collector.ts
      - name: Run SOC 2 control tests
        run: npx jest tests/soc2/ --ci

Vendor and Subprocessor Management

SOC 2 CC9.2 requires monitoring third-party risks. Every SaaS vendor that touches your data must be assessed.

# tests/soc2/test_vendor_compliance.py

REQUIRED_VENDOR_CERTS = {
    'AWS': ['SOC 2 Type II', 'ISO 27001'],
    'Stripe': ['SOC 2 Type II', 'PCI DSS Level 1'],
    'SendGrid': ['SOC 2 Type II'],
    'Datadog': ['SOC 2 Type II'],
}

VENDOR_CERT_FILE = 'compliance/vendor-certs.yaml'

def test_vendor_cert_file_exists():
    """SOC 2 CC9.2: Vendor compliance documentation must be maintained."""
    import os
    assert os.path.exists(VENDOR_CERT_FILE), (
        f"Vendor certification file not found: {VENDOR_CERT_FILE}. "
        f"Create and maintain this as part of your vendor management program."
    )

def test_vendor_certs_documented():
    """SOC 2 CC9.2: All key vendors have documented compliance certifications."""
    import yaml
    with open(VENDOR_CERT_FILE) as f:
        documented = yaml.safe_load(f)

    for vendor, required_certs in REQUIRED_VENDOR_CERTS.items():
        assert vendor in documented, f"Vendor {vendor} not documented in {VENDOR_CERT_FILE}"

        for cert in required_certs:
            assert cert in documented[vendor].get('certifications', []), (
                f"Vendor {vendor} missing required certification: {cert}"
            )

def test_vendor_certs_not_expired():
    """SOC 2 CC9.2: Vendor certifications must be current."""
    import yaml
    from datetime import datetime

    with open(VENDOR_CERT_FILE) as f:
        documented = yaml.safe_load(f)

    today = datetime.utcnow().date()

    for vendor, data in documented.items():
        cert_expiry = data.get('cert_expiry')
        if cert_expiry:
            expiry_date = datetime.strptime(cert_expiry, '%Y-%m-%d').date()
            assert expiry_date >= today, (
                f"Vendor {vendor} certification expired on {cert_expiry}. "
                f"Obtain updated certification report."
            )

From Testing to Audit Readiness

The gap between "we run these tests" and "we are SOC 2 compliant" is mostly documentation and consistency. Auditors want to see:

  1. Control documentation — what the control is, who owns it, how often it runs
  2. Evidence of consistent operation — test results, not just assertions that controls work
  3. Exception handling — what happened when a control failed, and how you responded

Automated testing covers the "consistent operation" column. Pair it with:

  • A control matrix mapping each TSC criterion to your specific controls
  • A ticketing workflow so test failures become tracked remediation tasks
  • Quarterly internal reviews before the auditor arrives

SOC 2 is achievable for engineering teams. The discipline it builds — access control, audit logging, change management, vendor oversight — makes your system more secure regardless of whether a customer ever requests your audit report.

Read more

Start now free