SOC 2

SOC 2 Evidence Collection Automation: Stop Emailing Screenshots to Your Auditor

HelpMeTest

03 Mar 2027 — 11 min read

SOC 2 audits are expensive not because testing is hard, but because evidence collection is a nightmare. Your auditor requests 200 pieces of evidence. You spend two weeks hunting down screenshots of Jira tickets, GitHub access logs, deployment records, and AWS Config snapshots. Then they come back with follow-up questions and you do it again.

The irony is that most of this evidence is already being generated automatically by your existing tools — GitHub Actions, AWS CloudTrail, Jira, PagerDuty. The problem is that it's scattered across a dozen systems and not in an audit-ready format.

This post covers how to build a continuous SOC 2 evidence collection pipeline: what auditors actually need, how to pull it automatically from your existing tools, and how to structure it so your next audit doesn't require a two-week fire drill.

What SOC 2 Auditors Actually Need

SOC 2 evidence falls into five Trust Service Criteria (TSC) categories. Here's what auditors look for in each, translated from the official guidance into concrete artifacts:

Security (CC6-CC9) — The One That Matters Most

Access reviews: Evidence that user access was reviewed quarterly. Needs timestamps, approver names, list of users reviewed.
Background checks: Evidence that employees completed background screening.
Security training: Completion records with employee names and completion dates.
Vulnerability scanning: Scan results showing you found and fixed vulnerabilities.
Penetration testing: Report from annual pentest with findings and remediation.
Logical access provisioning: Evidence that new accounts were created with appropriate approvals.
Termination procedures: Evidence that departed employee access was revoked promptly.
Change management: Evidence that production changes were reviewed and approved before deployment.

Availability (A1)

Uptime monitoring: Historical uptime data showing you met your SLA.
Incident records: Documentation of outages and response times.
Capacity planning: Evidence that you monitor and plan for capacity.
Backup testing: Evidence that backups were tested and restoration worked.

Confidentiality (C1)

Data classification: Evidence that data is classified and handled per policy.
Encryption at rest/transit: Configuration screenshots or certificates.
NDA records: Signed NDAs with employees and vendors.

What to Automate vs What to Do Manually

Not everything should be automated. Here's the breakdown:

Automate aggressively:

Deployment records (GitHub Actions artifacts)
Access logs (AWS CloudTrail, GitHub audit log)
Vulnerability scan results (Snyk, Trivy, AWS Inspector)
Configuration snapshots (AWS Config, Terraform state)
Uptime/availability metrics (your monitoring tool)
Test results (your CI pipeline)

Automate with tooling (Vanta/Drata/Secureframe):

User provisioning/deprovisioning events
MDM compliance status
Software inventory

Do manually but document systematically:

Quarterly access reviews (the approver judgment call can't be automated)
Penetration test commissioning
Vendor security assessments
Policy reviews and sign-offs

Building an Evidence Collection Pipeline from GitHub Actions

Every deployment that goes through GitHub Actions is evidence of change management. But "evidence" means more than the Actions log URL — it means a structured artifact that a non-engineer auditor can understand.

# .github/workflows/deploy-with-evidence.yml
name: Deploy with SOC 2 Evidence Collection

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Collect pre-deployment evidence
        id: pre-deploy
        run: |
          cat > /tmp/deployment-evidence.json << EOF
          {
            "deployment_id": "${{ github.run_id }}",
            "timestamp": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
            "initiator": "${{ github.actor }}",
            "commit_sha": "${{ github.sha }}",
            "commit_message": $(echo '${{ github.event.head_commit.message }}' | jq -R .),
            "repository": "${{ github.repository }}",
            "branch": "${{ github.ref_name }}",
            "environment": "production",
            "approvals": [],
            "test_results": null,
            "security_scan": null
          }
          EOF
      
      - name: Run tests and capture results
        id: tests
        run: |
          # Run your test suite
          npm test --reporter json > /tmp/test-results.json 2>&1 || true
          
          # Extract summary for evidence
          TEST_SUMMARY=$(cat /tmp/test-results.json | jq '{
            total: .numTotalTests,
            passed: .numPassedTests,
            failed: .numFailedTests,
            duration_ms: .testResults[0].perfStats.runtime,
            timestamp: .testResults[0].perfStats.end
          }' 2>/dev/null || echo '{"error": "test parsing failed"}')
          
          # Update evidence document
          jq --argjson tests "$TEST_SUMMARY" '.test_results = $tests' \
            /tmp/deployment-evidence.json > /tmp/deployment-evidence-updated.json
          mv /tmp/deployment-evidence-updated.json /tmp/deployment-evidence.json
      
      - name: Run security scan and capture results
        id: security-scan
        run: |
          # Run Trivy container scan
          trivy image --format json --output /tmp/trivy-results.json \
            ${{ env.DOCKER_IMAGE }} 2>/dev/null || true
          
          # Summarize for evidence
          VULN_SUMMARY=$(cat /tmp/trivy-results.json | jq '{
            critical: [.Results[].Vulnerabilities[]? | select(.Severity == "CRITICAL")] | length,
            high: [.Results[].Vulnerabilities[]? | select(.Severity == "HIGH")] | length,
            medium: [.Results[].Vulnerabilities[]? | select(.Severity == "MEDIUM")] | length,
            scan_timestamp: now | todate
          }' 2>/dev/null || echo '{"error": "scan parsing failed"}')
          
          jq --argjson scan "$VULN_SUMMARY" '.security_scan = $scan' \
            /tmp/deployment-evidence.json > /tmp/tmp.json
          mv /tmp/tmp.json /tmp/deployment-evidence.json
      
      # Production deployments require human approval (configured in GitHub Environments)
      - name: Capture approval evidence
        run: |
          # The fact that this step is running means the environment approval was granted
          # GitHub Actions records the approver — capture it
          APPROVAL_DATA=$(gh api \
            /repos/${{ github.repository }}/actions/runs/${{ github.run_id }}/approvals \
            --jq '[.[] | {approver: .user.login, approved_at: .submitted_at, state: .state}]' \
            2>/dev/null || echo '[]')
          
          jq --argjson approvals "$APPROVAL_DATA" '.approvals = $approvals' \
            /tmp/deployment-evidence.json > /tmp/tmp.json
          mv /tmp/tmp.json /tmp/deployment-evidence.json
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      
      - name: Deploy application
        run: |
          # Your actual deployment command here
          echo "Deploying..."
      
      - name: Finalize and store evidence
        if: always()
        run: |
          # Add deployment outcome
          DEPLOYMENT_STATUS="${{ job.status }}"
          jq --arg status "$DEPLOYMENT_STATUS" \
             --arg end_time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
             '.deployment_status = $status | .completed_at = $end_time' \
            /tmp/deployment-evidence.json > /tmp/tmp.json
          mv /tmp/tmp.json /tmp/deployment-evidence.json
          
          # Store in S3 with a predictable path for auditor access
          EVIDENCE_KEY="soc2-evidence/deployments/$(date +%Y/%m/%d)/${{ github.run_id }}.json"
          aws s3 cp /tmp/deployment-evidence.json "s3://${{ env.SOC2_EVIDENCE_BUCKET }}/$EVIDENCE_KEY"
          
          echo "Evidence stored at: s3://${{ env.SOC2_EVIDENCE_BUCKET }}/$EVIDENCE_KEY"
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.SOC2_EVIDENCE_AWS_KEY }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.SOC2_EVIDENCE_AWS_SECRET }}

Automating Access Review Evidence

Quarterly access reviews are one of the most labor-intensive SOC 2 requirements. You can't automate the review — a human has to make the judgment call about whether someone should still have access. But you can automate the evidence collection and evidence packaging.

#!/usr/bin/env python3
"""
SOC 2 Access Review Evidence Collector
Generates the access lists that need human review, then packages the reviewed results.

Run quarterly: ./collect-access-review.py --quarter 2024-Q1
"""

import json
import datetime
import boto3
import requests
from pathlib import Path

class AccessReviewEvidenceCollector:
    
    def __init__(self, output_dir: str = "soc2-evidence/access-reviews"):
        self.output_dir = Path(output_dir)
        self.output_dir.mkdir(parents=True, exist_ok=True)
        self.quarter = self._current_quarter()
    
    def _current_quarter(self) -> str:
        now = datetime.datetime.utcnow()
        quarter = (now.month - 1) // 3 + 1
        return f"{now.year}-Q{quarter}"
    
    def collect_aws_iam_users(self) -> list:
        """Get all IAM users with their last activity."""
        iam = boto3.client("iam")
        
        users = []
        paginator = iam.get_paginator("get_account_authorization_details")
        
        for page in paginator.paginate(Filter=["User"]):
            for user in page["UserDetailList"]:
                # Get password last used
                user_detail = iam.get_user(UserName=user["UserName"])["User"]
                
                # Get access key last used
                keys = iam.list_access_keys(UserName=user["UserName"])["AccessKeyMetadata"]
                key_last_used = None
                for key in keys:
                    if key["Status"] == "Active":
                        key_info = iam.get_access_key_last_used(AccessKeyId=key["AccessKeyId"])
                        last_used = key_info["AccessKeyLastUsed"].get("LastUsedDate")
                        if last_used:
                            key_last_used = last_used.isoformat()
                
                users.append({
                    "username": user["UserName"],
                    "user_id": user["UserId"],
                    "created_at": user["CreateDate"].isoformat(),
                    "password_last_used": user_detail.get("PasswordLastUsed", "Never").isoformat() 
                                         if hasattr(user_detail.get("PasswordLastUsed", ""), "isoformat") 
                                         else "Never",
                    "access_key_last_used": key_last_used,
                    "groups": [g["GroupName"] for g in user.get("GroupList", [])],
                    "has_mfa": len(iam.list_mfa_devices(UserName=user["UserName"])["MFADevices"]) > 0,
                    # Fields to be filled by reviewer
                    "review_status": None,  # "approved" | "revoke" | "modify"
                    "reviewed_by": None,
                    "reviewed_at": None,
                    "review_notes": None,
                })
        
        return users
    
    def collect_github_members(self, org: str, token: str) -> list:
        """Get all GitHub org members with their role and last activity."""
        headers = {"Authorization": f"token {token}", "Accept": "application/vnd.github+json"}
        
        members = []
        page = 1
        
        while True:
            response = requests.get(
                f"https://api.github.com/orgs/{org}/members",
                headers=headers,
                params={"per_page": 100, "page": page}
            )
            
            if not response.json():
                break
            
            for member in response.json():
                # Get member's role
                role_response = requests.get(
                    f"https://api.github.com/orgs/{org}/memberships/{member['login']}",
                    headers=headers
                )
                role = role_response.json().get("role", "member")
                
                members.append({
                    "username": member["login"],
                    "github_id": member["id"],
                    "role": role,
                    "profile_url": member["html_url"],
                    # Fields to be filled by reviewer
                    "review_status": None,
                    "reviewed_by": None,
                    "reviewed_at": None,
                })
            
            page += 1
        
        return members
    
    def generate_review_package(self, aws_users: list, github_members: list) -> str:
        """Generate the access review package for human review."""
        timestamp = datetime.datetime.utcnow().isoformat()
        
        package = {
            "review_id": f"access-review-{self.quarter}",
            "quarter": self.quarter,
            "generated_at": timestamp,
            "status": "pending_review",
            "reviewer": None,
            "review_completed_at": None,
            "systems": {
                "aws_iam": {
                    "system": "AWS IAM",
                    "collected_at": timestamp,
                    "total_users": len(aws_users),
                    "users_requiring_attention": [
                        u for u in aws_users 
                        if u.get("password_last_used") == "Never" 
                        or not u.get("has_mfa")
                    ],
                    "users": aws_users,
                },
                "github": {
                    "system": "GitHub Organization",
                    "collected_at": timestamp,
                    "total_members": len(github_members),
                    "members": github_members,
                }
            },
            "instructions": (
                "Review each user. For each: set review_status to 'approved', 'revoke', or 'modify'. "
                "Set reviewed_by to your name and reviewed_at to the current timestamp. "
                "After completing review, set top-level status to 'completed'."
            )
        }
        
        output_file = self.output_dir / f"access-review-{self.quarter}.json"
        
        with open(output_file, "w") as f:
            json.dump(package, f, indent=2, default=str)
        
        print(f"Access review package generated: {output_file}")
        print(f"Users requiring attention:")
        print(f"  AWS IAM: {len(package['systems']['aws_iam']['users_requiring_attention'])} users (no MFA or never logged in)")
        
        return str(output_file)
    
    def validate_completed_review(self, review_file: str) -> bool:
        """
        Validate that a completed access review has all required fields.
        Run this before uploading to your evidence store.
        """
        with open(review_file) as f:
            review = json.load(f)
        
        errors = []
        
        if review.get("status") != "completed":
            errors.append("Review status is not 'completed'")
        
        if not review.get("reviewer"):
            errors.append("Reviewer not recorded")
        
        if not review.get("review_completed_at"):
            errors.append("Review completion timestamp not recorded")
        
        # Check that all users have been reviewed
        for system_name, system in review.get("systems", {}).items():
            users = system.get("users", system.get("members", []))
            unreviewed = [u for u in users if u.get("review_status") is None]
            if unreviewed:
                errors.append(
                    f"{system_name}: {len(unreviewed)} users not reviewed: "
                    f"{[u.get('username') for u in unreviewed[:5]]}"
                )
        
        if errors:
            print("Access review validation FAILED:")
            for error in errors:
                print(f"  - {error}")
            return False
        
        print(f"Access review validation passed. Ready to upload as SOC 2 evidence.")
        return True

DIY vs Vanta/Drata/Secureframe

The compliance SaaS tools (Vanta at ~$800-1500/month, Drata at similar pricing, Secureframe at ~$600-1200/month) automate a significant portion of evidence collection. Here's an honest comparison:

Where they add real value:

MDM compliance monitoring (they integrate with Jamf/Intune directly)
Employee onboarding/offboarding checklists with automatic evidence
Vendor questionnaire tracking
Pre-built auditor access portal (auditors can log in and pull evidence themselves)
Automated control testing for common controls

Where DIY is better:

Custom evidence types specific to your stack
Custom controls that don't fit their templates
Evidence that requires custom API integrations they don't support
If your stack is AWS-heavy with heavy Terraform usage (write your own scripts)

The honest answer: If you're doing SOC 2 for the first time and have fewer than 50 employees, Vanta or Drata saves you more than they cost. If you're doing your third renewal and have a mature DevOps practice, DIY automation is more flexible and cheaper.

Storing Evidence in Audit-Ready Format

Evidence stored as individual files in an S3 bucket is not audit-ready. Auditors need to see:

What the evidence is (context, not just a raw JSON dump)
When it was collected
Who collected it
What control it satisfies

class SOC2EvidenceStore:
    """
    Store SOC 2 evidence with the metadata auditors need to use it.
    Each piece of evidence gets a manifest with control mapping.
    """
    
    CONTROL_MAPPING = {
        "deployment-record": {
            "trust_service_criteria": "CC8.1",
            "description": "Change management — changes deployed to production via approved pipeline",
            "evidence_type": "system_generated",
            "population": "All production deployments",
        },
        "access-review": {
            "trust_service_criteria": "CC6.2",
            "description": "Logical access — user access reviewed quarterly by management",
            "evidence_type": "management_review",
            "frequency": "quarterly",
        },
        "vulnerability-scan": {
            "trust_service_criteria": "CC7.1",
            "description": "System monitoring — vulnerabilities identified and tracked",
            "evidence_type": "system_generated",
            "frequency": "continuous",
        },
        "uptime-report": {
            "trust_service_criteria": "A1.2",
            "description": "Availability — system availability monitored against SLA",
            "evidence_type": "system_generated",
            "frequency": "monthly",
        },
    }
    
    def __init__(self, s3_bucket: str, aws_region: str = "us-east-1"):
        self.s3 = boto3.client("s3", region_name=aws_region)
        self.bucket = s3_bucket
    
    def store_evidence(
        self, 
        evidence_type: str,
        evidence_data: dict,
        period: str,  # e.g., "2024-Q1" or "2024-01"
        collected_by: str = "automated-pipeline"
    ) -> str:
        """
        Store evidence with full audit metadata.
        Returns the S3 URI of the stored evidence.
        """
        control_info = self.CONTROL_MAPPING.get(evidence_type, {})
        
        evidence_package = {
            "metadata": {
                "evidence_type": evidence_type,
                "evidence_id": f"{evidence_type}-{period}-{datetime.datetime.utcnow().strftime('%Y%m%dT%H%M%S')}",
                "period": period,
                "collected_at": datetime.datetime.utcnow().isoformat(),
                "collected_by": collected_by,
                "trust_service_criteria": control_info.get("trust_service_criteria"),
                "control_description": control_info.get("description"),
                "evidence_type_label": control_info.get("evidence_type"),
            },
            "evidence": evidence_data,
        }
        
        key = f"soc2-evidence/{evidence_type}/{period}/{evidence_package['metadata']['evidence_id']}.json"
        
        self.s3.put_object(
            Bucket=self.bucket,
            Key=key,
            Body=json.dumps(evidence_package, indent=2, default=str),
            ContentType="application/json",
            ServerSideEncryption="AES256",
            Metadata={
                "evidence-type": evidence_type,
                "period": period,
                "tsc": control_info.get("trust_service_criteria", "unknown"),
            }
        )
        
        return f"s3://{self.bucket}/{key}"
    
    def generate_auditor_index(self, audit_period_start: str, audit_period_end: str) -> dict:
        """
        Generate an index of all evidence collected for a given audit period.
        This is what you hand to your auditor at the start of an engagement.
        """
        evidence_index = {}
        
        paginator = self.s3.get_paginator("list_objects_v2")
        
        for page in paginator.paginate(Bucket=self.bucket, Prefix="soc2-evidence/"):
            for obj in page.get("Contents", []):
                # Get metadata
                head = self.s3.head_object(Bucket=self.bucket, Key=obj["Key"])
                metadata = head.get("Metadata", {})
                
                tsc = metadata.get("tsc", "unknown")
                if tsc not in evidence_index:
                    evidence_index[tsc] = []
                
                evidence_index[tsc].append({
                    "evidence_id": obj["Key"].split("/")[-1].replace(".json", ""),
                    "s3_key": obj["Key"],
                    "collected_at": obj["LastModified"].isoformat(),
                    "evidence_type": metadata.get("evidence-type"),
                    "period": metadata.get("period"),
                    "size_bytes": obj["Size"],
                })
        
        return {
            "audit_period": {
                "start": audit_period_start,
                "end": audit_period_end,
            },
            "generated_at": datetime.datetime.utcnow().isoformat(),
            "evidence_by_control": evidence_index,
            "total_evidence_items": sum(len(v) for v in evidence_index.values()),
        }

Testing That Controls Are Actually Working

Evidence collection proves that data was gathered. Control testing proves that the controls are actually effective. These are different things. Here's how to test key SOC 2 controls continuously:

class TestSOC2Controls:
    
    def test_production_deployment_requires_approval(self, github_client):
        """
        CC8.1: Changes to production must be reviewed and approved before deployment.
        Test that the GitHub branch protection rules are configured.
        """
        protection = github_client.get(
            f"/repos/{ORG}/{REPO}/branches/main/protection"
        ).json()
        
        # Must require pull request reviews
        required_reviews = protection.get("required_pull_request_reviews", {})
        assert required_reviews.get("required_approving_review_count", 0) >= 1, \
            "Production branch does not require at least 1 approving review"
        
        # Must dismiss stale reviews when new commits are pushed
        assert required_reviews.get("dismiss_stale_reviews") == True, \
            "Stale reviews not dismissed — old approvals persist after new commits"
        
        # Must require status checks to pass
        status_checks = protection.get("required_status_checks", {})
        assert status_checks.get("strict") == True, \
            "Branch not required to be up to date before merging"
        
        # Admins must not be able to bypass protections
        assert protection.get("enforce_admins", {}).get("enabled") == True, \
            "Branch protection rules do not apply to admins — bypass risk"
    
    def test_mfa_enforced_for_all_users(self, aws_iam_client):
        """
        CC6.1: Multi-factor authentication required for all users with console access.
        """
        iam = boto3.client("iam")
        
        # Get all users
        users = iam.list_users()["Users"]
        
        users_without_mfa = []
        for user in users:
            # Check if user has console access (login profile)
            try:
                iam.get_login_profile(UserName=user["UserName"])
                has_console_access = True
            except iam.exceptions.NoSuchEntityException:
                has_console_access = False
            
            if has_console_access:
                mfa_devices = iam.list_mfa_devices(UserName=user["UserName"])["MFADevices"]
                if not mfa_devices:
                    users_without_mfa.append(user["UserName"])
        
        assert len(users_without_mfa) == 0, (
            f"The following users have console access without MFA: {users_without_mfa}. "
            f"SOC 2 CC6.1 requires MFA for all users with access to systems containing sensitive data."
        )
    
    def test_encryption_at_rest_enabled(self, aws_client):
        """
        C1.1: All data at rest must be encrypted.
        Check that S3 buckets, RDS instances, and EBS volumes are encrypted.
        """
        s3 = boto3.client("s3")
        rds = boto3.client("rds")
        
        unencrypted = []
        
        # Check S3 buckets
        for bucket in s3.list_buckets()["Buckets"]:
            try:
                enc = s3.get_bucket_encryption(Bucket=bucket["Name"])
                rules = enc["ServerSideEncryptionConfiguration"]["Rules"]
                if not rules:
                    unencrypted.append(f"S3:{bucket['Name']} (no encryption)")
            except s3.exceptions.ServerSideEncryptionConfigurationNotFoundError:
                unencrypted.append(f"S3:{bucket['Name']} (no encryption configured)")
        
        # Check RDS instances
        for instance in rds.describe_db_instances()["DBInstances"]:
            if not instance.get("StorageEncrypted"):
                unencrypted.append(f"RDS:{instance['DBInstanceIdentifier']} (not encrypted)")
        
        assert len(unencrypted) == 0, (
            f"Unencrypted data stores found: {unencrypted}. "
            f"All data at rest must be encrypted for SOC 2 C1.1."
        )

Continuous SOC 2 Control Testing with HelpMeTest

The difference between a SOC 2 Type 1 (point-in-time snapshot) and Type 2 (operating effectiveness over a period) is evidence that controls worked continuously — not just when you ran a check for the auditor.

HelpMeTest gives you continuous control validation through browser-based tests that run 24/7:

Test: SOC 2 — Production Deployment Requires Approval
Schedule: Daily at 09:00 UTC

Steps:
1. Navigate to https://github.com/{org}/{repo}/settings/branches
2. Log in as admin user (using saved "GitHub Admin" session)
3. Click on the "main" branch protection rule
4. Verify "Require a pull request before merging" is checked
5. Verify "Required approvals" shows 1 or more
6. Verify "Dismiss stale pull request approvals when new commits are pushed" is checked
7. Verify "Require status checks to pass before merging" is checked
8. Verify "Include administrators" is checked
9. Take a screenshot as evidence
10. Navigate to https://github.com/{org}/{repo}/actions
11. Verify the last 5 deployments all show a green check (successful tests before deploy)

This test runs daily and produces timestamped screenshots showing your controls are continuously enforced — exactly what a SOC 2 Type 2 auditor needs to see.

At $100/month, HelpMeTest generates continuous control evidence that costs thousands of dollars per year to collect manually. More importantly, it tells you immediately when a control breaks — before the auditor notices.

Key Takeaways

Automating SOC 2 evidence collection is not about eliminating the audit burden — it's about shifting from reactive (two-week fire drill) to continuous (always audit-ready):

Instrument your CI/CD pipeline — every deployment should auto-generate structured evidence with approver info, test results, and security scan results.
Automate access data collection — don't collect lists manually; generate them from your authoritative systems.
Separate data collection from human review — automate the data gathering, but keep humans in the loop for judgment calls.
Store evidence with control mapping — raw data without context is not useful to auditors.
Test that controls work, not just that evidence exists — a passing test proves effectiveness; a screenshot proves someone looked at it.
Run control tests continuously — SOC 2 Type 2 requires evidence of continuous effectiveness, not a single snapshot.

The goal is not to spend less time on SOC 2. It's to move SOC 2 compliance from a periodic audit event to a continuous operational practice.