Continuous Testing for Trunk-Based Development: Keeping Main Green

Trunk-based development (TBD) requires one thing above all else: main must always be deployable. A broken main blocks everyone. In a team of 10 developers committing multiple times per day, a broken main is catastrophic.

Continuous testing is what makes TBD viable at scale. The question isn't "do we test before merge?" — it's "what tests, how fast, with what gate conditions?"

The Core Constraint: Feedback Must Be Fast

In long-lived feature branch workflows, CI can take 30 minutes and it's acceptable — developers merge infrequently. In TBD, developers commit multiple times per day. A 30-minute CI gate means a developer waits 30 minutes between commits.

The feedback loop must be fast enough that waiting for CI doesn't interrupt flow. Target: < 5 minutes for the blocking gate.

This creates a fundamental tension: thorough testing takes time; fast feedback requires shortcuts. The resolution is tiered testing.

Tiered Testing Architecture

Split the test suite into gates with different time and coverage profiles:

Tier 1 — Pre-commit (local, < 30 seconds)

• Unit tests for changed modules only
• Linting and formatting
• Type checking

This tier runs locally before push. Not enforced by CI (would be too slow), but committed developers run it.

Tier 2 — Pre-merge gate (CI, < 5 minutes)

• Changed module unit tests
• Integration tests for directly affected services
• Critical path smoke tests

This tier blocks merge. It must be fast. Change-based test selection is essential here — running only tests affected by the specific change.

Tier 3 — Post-merge validation (CI, < 20 minutes)

• Full test suite
• Performance tests
• Cross-service integration tests

This tier runs after merge, against main. It doesn't block the committer (they've already merged) but pages the team if it fails.

Tier 4 — Scheduled (daily/nightly, < 60 minutes)

• E2E tests
• Security scans
• Load tests
• Mutation testing

This tier runs on a schedule, not per-commit. Failures are treated as P1 bugs to fix the next morning.

Pre-Merge Gate Design

The pre-merge gate is the most important tier. It must catch breaking changes without being slow enough to frustrate developers.

What belongs in the pre-merge gate

✅ Include:

• Fast unit tests for changed code
• Integration tests that verify changed behavior
• Smoke tests for critical user flows
• Static analysis (linting, type checking, security scanning)
• Dependency vulnerability checks

❌ Exclude:

• Slow E2E tests (move to post-merge or scheduled)
• Tests unrelated to the change (move to post-merge)
• Performance benchmarks (too variable for fast CI)
• Visual regression tests (move to scheduled or manual)

Gate performance targets

If you're not hitting these targets, investigate: caching (npm install shouldn't take 2 minutes), change-based selection (running all 10,000 unit tests for a one-line change), or parallelization (sequential instead of parallel execution).

Handling gate failures

When the pre-merge gate fails:

1. Developer fixes before merging — non-negotiable. Don't merge broken code.
2. Gate failure is visible in PR — GitHub/GitLab checks integrate automatically.
3. Fast fix path — developer can re-trigger only the failed step, not the whole gate.
4. Flaky failures — one flaky failure shouldn't block merge. Auto-retry once; if it fails twice, it's real.

The "one retry" policy is important. Zero retries → flaky gates block development. Unlimited retries → real failures get silently retried away.

Post-Merge Validation

Post-merge validation runs after merge but before or concurrent with deployment. It catches what the fast pre-merge gate missed.

Configuration

# GitHub Actions: post-merge validation
on:
  push:
    branches: [main]

jobs:
  full-validation:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
      - name: Run full test suite
        run: npm test -- --coverage
      - name: Performance regression check
        run: npm run bench -- --ci
      - name: Notify on failure
        if: failure()
        uses: actions/github-script@v7
        with:
          script: |
            github.rest.issues.create({
              owner: context.repo.owner,
              repo: context.repo.repo,
              title: `Post-merge CI failure on main`,
              body: `Run: ${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`,
              labels: ['ci-failure', 'high-priority']
            })

Post-merge failures create GitHub issues automatically. Someone is responsible for fixing main, and the failure is tracked.

Who owns a post-merge failure?

The committer who broke main is responsible for fixing it. This requires knowing who broke it — git bisect or CI tooling that associates failures with commits.

Some teams use an "on-call for main" rotation: one engineer per week is responsible for fixing any main breakage regardless of who introduced it. This reduces context-switching for individual developers and creates incentive to keep main healthy.

Feature Flags as a Testing Strategy

In TBD, incomplete features can't live in separate branches. Feature flags are the solution: deploy incomplete code behind a flag, test it safely, enable for users when ready.

Testing with feature flags

Every feature flag creates test variants: tests that run with the flag on, and tests that run with the flag off.

# Test matrix with feature flags
@pytest.mark.parametrize("new_checkout_enabled", [True, False])
def test_checkout_flow(new_checkout_enabled):
    with feature_flag("new_checkout", enabled=new_checkout_enabled):
        result = checkout(cart)
        
        if new_checkout_enabled:
            assert result.payment_processor == "stripe_v2"
        else:
            assert result.payment_processor == "stripe_v1"

This pattern ensures the old code path still works while the new one is being built.

Flag lifecycle testing

Feature flags create technical debt. A flag left enabled for 18 months with no tests for the disabled path becomes untested dead code.

Track flag age and test coverage for both flag states:

# Fail if any feature flag is > 90 days old and still in code
def test_no_stale_feature_flags():
    stale_flags = [
        flag for flag in get_all_feature_flags()
        if flag.age_days > 90 and not flag.permanent
    ]
    assert not stale_flags, f"Stale flags need cleanup: {stale_flags}"

Progressive rollout testing

Feature flags enable progressive rollout: enable for 1% of users, then 10%, then 100%. Testing at each rollout stage:

Pre-1% rollout: Unit and integration tests, internal dogfooding

1% rollout: Monitor error rates, latency, user-facing metrics. Automated comparison against control group.

10% rollout: Same monitoring, higher signal volume. Canary analysis comparing flag=on cohort vs. flag=off cohort.

Full rollout: Remove the flag from code (flagless = the feature is the product).

The rollout testing strategy connects CI testing with production observability — each stage has explicit test criteria before proceeding to the next.

Managing Test Flakiness at Merge Frequency

High merge frequency amplifies flakiness. If you merge 20 times per day with a 5% per-run flakiness rate, you'll see a flaky failure almost every merge. This erodes trust in CI and trains developers to ignore red signals.

Flakiness at scale

At 20 merges/day with 1,000 tests at 0.1% flakiness each:

Expected flaky failures per merge = 1,000 × 0.001 = 1 flaky test per merge

At 20 merges/day, this is 20 flaky failures per day, each requiring investigation. The CI system becomes noise.

Zero flakiness tolerance in the pre-merge gate is the goal. Every flaky test is a CI incident.

Flakiness detection in TBD

TBD environments produce more data for flakiness detection (more runs per day) while having less tolerance for it. Use the volume to your advantage:

# Any test failing more than N times per day without code changes = quarantine
def detect_newly_flaky_tests(today_failures, recent_code_changes):
    return [
        test for test in today_failures
        if not is_affected_by_changes(test, recent_code_changes)
        and today_failures[test] > 2
    ]

Auto-quarantine tests that fail multiple times in one day without code changes. Review quarantined tests at end of week.

Deployment Pipeline Integration

In TBD, CI testing connects directly to deployment. The pipeline:

commit → pre-merge gate → merge → post-merge validation → staging deployment → smoke tests → production deployment

At each stage, test failure stops the pipeline:

• Pre-merge gate failure → developer fixes before merge
• Post-merge validation failure → open incident, revert if P0
• Staging smoke test failure → deployment blocked, developer investigates
• Production smoke test failure → rollback triggered

The continuous testing model means any failure is actionable immediately — not discovered days later.

Automated rollback testing

When rollback is automated, test the rollback itself. A rollback that fails during an incident is a secondary disaster.

# Verify rollback works in staging on every deploy
- name: Test rollback procedure
  run: |
    # Deploy current version
    kubectl apply -f deploy/staging.yaml
    
    # Verify deployment
    ./scripts/smoke-test.sh staging
    
    # Roll back to previous version
    kubectl rollout undo deployment/app
    
    # Verify rollback succeeded
    ./scripts/smoke-test.sh staging

Rollback tests confirm both the forward deployment and the reverse work correctly. Discovering that rollback is broken during a production incident is too late.

Summary

Continuous testing for trunk-based development requires a tiered architecture: a fast pre-merge gate that catches breaking changes without blocking flow, post-merge validation that covers what speed required omitting, and scheduled runs for expensive tests.

The critical discipline: the pre-merge gate must be fast (< 5 minutes) and reliable (near-zero false positives). Flaky gates and slow gates both erode trust in CI, and developers who don't trust CI merge without waiting.

Feature flags enable incomplete code to be deployed safely. Progressive rollout connects CI testing with production observability. Together, they make trunk-based development work at scale.