Deployment Verification Testing: The Complete Guide
Deployment verification testing (DVT) is the set of checks you run immediately after a deployment completes to confirm the new version is working correctly in the target environment. It bridges the gap between "the deployment pipeline succeeded" and "the application is actually working."
A successful CI pipeline tells you the code compiled, unit tests passed, and the image was pushed to the registry. Deployment verification testing tells you the application is alive, connected to its dependencies, and capable of serving real requests.
The Deployment Verification Gap
Most teams have strong pre-deployment testing — unit tests, integration tests, staging environments. The gap appears after the deployment command runs and before the team declares the release complete.
In this window:
- The new container may start but fail to connect to the database
- An environment variable may be missing or misconfigured in production
- A dependency service may have changed its API contract
- SSL certificates may have expired or not yet propagated
- A feature flag may be in the wrong state
These failures don't appear in CI because they're environment-specific. Deployment verification testing is specifically designed to catch them.
Layers of Deployment Verification
Effective deployment verification works in layers, from fastest to most thorough:
Layer 1: Infrastructure checks (< 30 seconds)
- Deployment rollout completed without errors
- All pods/instances are running and healthy
- Health endpoint returns 200
Layer 2: Smoke tests (< 5 minutes)
- Core user flows work end-to-end
- Authentication succeeds
- Primary feature is functional
Layer 3: Synthetic monitoring (ongoing)
- Scheduled tests running against production continuously
- Alert within minutes if anything degrades
Layer 4: Canary analysis (< 30 minutes)
- Error rates of new version vs. old version compared
- Latency percentiles compared
- Business metrics (conversion, signups) compared
Run all four layers for significant deployments. For minor deployments (config updates, copy changes), Layer 1 and Layer 2 may be sufficient.
Implementing Layer 1: Infrastructure Checks
#!/bin/bash
# verify-infrastructure.sh
SERVICE=$1
NAMESPACE=${2:-production}
EXPECTED_IMAGE=$3
echo "=== Infrastructure Verification ==="
# Check rollout status
echo "Checking rollout..."
kubectl rollout status deployment/$SERVICE -n $NAMESPACE --timeout=120s || {
echo "FAIL: Rollout did not complete"
exit 1
}
# Check image
ACTUAL_IMAGE=$(kubectl get deployment $SERVICE -n $NAMESPACE \
-o jsonpath='{.spec.template.spec.containers[0].image}')
[ "$ACTUAL_IMAGE" = "$EXPECTED_IMAGE" ] || {
echo "FAIL: Image mismatch. Expected $EXPECTED_IMAGE, got $ACTUAL_IMAGE"
exit 1
}
# Check replica health
READY=$(kubectl get deployment $SERVICE -n $NAMESPACE \
-o jsonpath='{.status.readyReplicas}')
DESIRED=$(kubectl get deployment $SERVICE -n $NAMESPACE \
-o jsonpath='{.status.replicas}')
[ "$READY" = "$DESIRED" ] || {
echo "FAIL: Only $READY/$DESIRED replicas ready"
exit 1
}
# Check health endpoint
STATUS=$(curl -s -o /dev/null -w "%{http_code}" https://app.example.com/health)
[ "$STATUS" = "200" ] || {
echo "FAIL: Health check returned HTTP $STATUS"
exit 1
}
echo "PASS: Infrastructure checks passed"Implementing Layer 2: Smoke Tests
Smoke tests verify the application's critical paths. Write them once, run them on every deployment:
*** Settings ***
Library Browser
Suite Setup New Browser headless=True
Suite Teardown Close Browser
*** Variables ***
${BASE_URL} ${ENV_URL}
*** Test Cases ***
Application Is Reachable
New Page ${BASE_URL}
Get Title contains MyApp
User Authentication Works
New Page ${BASE_URL}/login
Fill Text [name="email"] deploy-test@example.com
Fill Text [name="password"] ${DEPLOY_TEST_PASSWORD}
Click button[type="submit"]
Wait For URL **/*dashboard*
Core Feature Is Functional
Click [data-testid="create-resource"]
Fill Text [name="name"] Deploy Verification Test
Click button[type="submit"]
Get Text .toast-success contains Created successfully
Database Connectivity Confirmed
# If the app can create and retrieve data, DB is connected
${url}= Get URL
Go To ${BASE_URL}/api/resources?created_by=deploy-test
${response}= Get Text pre
Should Contain ${response} Deploy Verification Test
Third-party Integrations Responding
${response}= HTTP GET ${BASE_URL}/api/status/integrations
${body}= Evaluate json.loads('''${response.body}''')
Should Be Equal ${body['stripe']['status']} connected
Should Be Equal ${body['sendgrid']['status']} connectedImplementing Layer 3: Continuous Synthetic Monitoring
HelpMeTest runs your smoke tests on a schedule — every 5 minutes by default, every 10 seconds on Enterprise — and alerts you the moment they fail.
Setup:
- Write your smoke tests in HelpMeTest
- Set monitoring interval to match your alerting requirements
- Configure Slack or email alerts for immediate notification
After a deployment, your verification window becomes the time between deployment and first alert. If HelpMeTest runs every 5 minutes and your deployment takes 3 minutes to roll out, you'll know within 8 minutes if something is wrong.
Implementing Layer 4: Canary Analysis
Compare the new version against the old version on real production traffic before completing the rollout:
# Flagger canary configuration
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
name: my-app
spec:
analysis:
interval: 1m
threshold: 5 # Allow 5 failed checks before rollback
maxWeight: 50 # Cap canary at 50% of traffic
stepWeight: 10 # Increment 10% at a time
metrics:
- name: request-success-rate
min: 99 # Rollback if success rate drops below 99%
interval: 1m
- name: request-duration
max: 500 # Rollback if P99 latency exceeds 500ms
interval: 1mCanary analysis gives you quantitative pass/fail criteria for the new version based on real traffic, not synthetic tests.
Defining Verification Pass/Fail Criteria
Document what "verified" means before the deployment, not after:
# Deployment Verification Criteria — my-app v2.0.0
## Must Pass (rollback if any fail)
- [ ] All pods healthy and running
- [ ] Health endpoint returns 200
- [ ] Smoke test suite completes in < 5 minutes with 0 failures
- [ ] Error rate < 0.1% (5-minute window post-deploy)
- [ ] P99 latency < 400ms (vs. baseline 280ms)
## Observation Period (30 minutes)
- [ ] No error rate spike > 2x baseline
- [ ] No latency degradation > 50% over baseline
- [ ] No alerts from monitoring
- [ ] No reports from customer support
## Success Criteria
All "Must Pass" items pass + 30-minute observation clean
Declared complete by: on-call engineerHaving written criteria prevents the "it looks fine, let's call it done" declaration that leaves a slow degradation unnoticed.
Automating the Full Verification Workflow
# .github/workflows/deploy-verify.yml
name: Deploy and Verify
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Deploy to production
run: |
kubectl set image deployment/my-app \
app=my-app:${{ github.sha }} \
-n production
kubectl rollout status deployment/my-app -n production
verify-infrastructure:
needs: deploy
runs-on: ubuntu-latest
steps:
- name: Infrastructure checks
run: ./scripts/verify-infrastructure.sh my-app production my-app:${{ github.sha }}
smoke-tests:
needs: verify-infrastructure
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install Robot Framework
run: pip install robotframework robotframework-browser
- name: Run smoke suite
env:
ENV_URL: ${{ vars.PRODUCTION_URL }}
DEPLOY_TEST_PASSWORD: ${{ secrets.DEPLOY_TEST_PASSWORD }}
run: robot smoke/
- name: Rollback on failure
if: failure()
run: kubectl rollout undo deployment/my-app -n production
notify:
needs: smoke-tests
runs-on: ubuntu-latest
if: success()
steps:
- name: Notify success
run: |
curl -X POST ${{ secrets.SLACK_WEBHOOK }} \
-d '{"text": "✅ my-app ${{ github.sha }} deployed and verified"}'Common Verification Failures and What They Mean
| Failure | Likely Cause | Resolution |
|---|---|---|
| Health check 503 immediately | App crashing on startup | Check pod logs: kubectl logs -l app=my-app --previous |
| Health check 200 but smoke tests fail | App alive but misconfigured | Check env vars, feature flags |
| Smoke tests pass but error rate rising | Load-dependent bug | Extend observation window, prepare rollback |
| Latency increase after deploy | New query or cache miss | Profile, consider rollback |
| Intermittent smoke test failures | Flaky test or race condition | Re-run once; if fails again, rollback |
Summary
Deployment verification testing is what happens between "deployment pipeline succeeded" and "release is complete." Run infrastructure checks to confirm the new version is actually running. Run smoke tests to confirm critical paths work. Monitor continuously to catch degradation that only appears under real traffic. Define pass/fail criteria before the deployment, not after.
The goal is a deployment that's declared complete based on evidence — not optimism.
