GitLab Environments and Deployment Testing: Review Apps, QA Environments, and Rollback

GitLab Environments and Deployment Testing: Review Apps, QA Environments, and Rollback

Deploying software isn't a single event — it's a series of progressively higher-stakes environments, each with different testing requirements. Code that passes unit tests in CI needs to prove itself in integration environments, staging, and finally production. GitLab's environments feature tracks these deployments and provides tools for managing the promotion process.

This guide covers GitLab environments from a testing perspective: how to configure environment-specific test jobs, use review apps for pre-merge QA, validate deployments automatically, and implement rollback-tested releases.

Understanding GitLab Environments

In GitLab, an environment represents a deployment target with a name, URL, and deployment history. Each deployment to an environment is tracked with:

  • The commit SHA deployed
  • When the deployment occurred
  • Who triggered it
  • Whether it succeeded or failed
  • The pipeline that created it

The Environments page in your project shows the current deployed version of each environment, with links to the deployed URL and the ability to roll back to a previous deployment.

# Basic environment configuration
deploy-staging:
  stage: deploy
  script:
    - ./deploy.sh staging
  environment:
    name: staging
    url: https://staging.yourapp.com

Environment Tiers

GitLab allows classifying environments by tier, which affects how they're displayed and what automation applies:

deploy-development:
  environment:
    name: development
    url: https://dev.yourapp.com
    deployment_tier: development

deploy-staging:
  environment:
    name: staging
    url: https://staging.yourapp.com
    deployment_tier: staging

deploy-production:
  environment:
    name: production
    url: https://yourapp.com
    deployment_tier: production

Available tiers: production, staging, testing, development, other.

Environment-Specific Test Jobs

Different environments require different tests. Smoke tests that run after deployment verify the deployment itself. Full regression suites run before promotion. Production has read-only health checks that don't mutate data.

stages:
  - deploy
  - verify

deploy-staging:
  stage: deploy
  script:
    - ./deploy.sh staging
  environment:
    name: staging
    url: https://staging.yourapp.com

smoke-test-staging:
  stage: verify
  script:
    - npx playwright test tests/smoke/ --project=chromium
  variables:
    BASE_URL: https://staging.yourapp.com
  environment:
    name: staging
    action: verify
  needs:
    - deploy-staging

regression-test-staging:
  stage: verify
  parallel: 4
  script:
    - npx playwright test tests/regression/ --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  variables:
    BASE_URL: https://staging.yourapp.com
  needs:
    - smoke-test-staging  # Only run full regression after smoke passes
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'

The action: verify on smoke-test-staging links the test job to the staging environment in GitLab's UI, creating a clear connection between deployments and their verification runs.

Review Apps for QA

Review apps provide dedicated environments for each merge request, enabling QA testing before code merges. Each MR gets its own URL, and the environment is destroyed when the MR is closed.

deploy-review:
  stage: deploy
  script:
    - |
      # Example using Heroku Review Apps
      heroku create review-$CI_ENVIRONMENT_SLUG --remote review
      git push review HEAD:main
      heroku run rake db:migrate --remote review
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    url: https://review-$CI_ENVIRONMENT_SLUG.herokuapp.com
    on_stop: stop-review
    auto_stop_in: 2 days
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
  
stop-review:
  stage: deploy
  script:
    - heroku destroy review-$CI_ENVIRONMENT_SLUG --confirm review-$CI_ENVIRONMENT_SLUG
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    action: stop
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      when: manual

The auto_stop_in: 2 days parameter automatically stops the review app if the MR goes unmerged, preventing environment sprawl.

Dynamic review app URLs with Kubernetes:

deploy-review:
  stage: deploy
  image: bitnami/kubectl
  script:
    - |
      export NAMESPACE=review-${CI_ENVIRONMENT_SLUG}
      kubectl create namespace $NAMESPACE --dry-run=client -o yaml | kubectl apply -f -
      
      # Substitute environment variables in manifests
      envsubst < k8s/review-app.yaml | kubectl apply -n $NAMESPACE -f -
      
      # Wait for deployment to be ready
      kubectl rollout status deployment/app -n $NAMESPACE --timeout=120s
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    url: https://$CI_ENVIRONMENT_SLUG.review.k8s.yourapp.com
    on_stop: stop-review
    kubernetes:
      namespace: review-${CI_ENVIRONMENT_SLUG}

Post-Deployment Verification

After deploying to any environment, automated verification tests confirm the deployment is healthy. These differ from pre-merge tests — they test the running system, not the code in isolation.

Smoke tests (fast, critical path only):

#!/bin/bash
# scripts/smoke-test.sh

BASE_URL=$1
PASS=0
FAIL=0

check() {
  local url="$BASE_URL$1"
  local expected=$2
  local actual=$(curl -s -o /dev/null -w "%{http_code}" "$url")
  
  if [ "$actual" == "$expected" ]; then
    echo "PASS: GET $url -> $actual"
    PASS=$((PASS+1))
  else
    echo "FAIL: GET $url -> $actual (expected $expected)"
    FAIL=$((FAIL+1))
  fi
}

check "/" "200"
check "/health" "200"
check "/api/v1/status" "200"
check "/nonexistent-page" "404"
check "/api/v1/protected" "401"

echo ""
echo "Results: $PASS passed, $FAIL failed"
[ $FAIL -eq 0 ] || exit 1
smoke-test:
  stage: verify
  script:
    - bash scripts/smoke-test.sh $CI_ENVIRONMENT_URL
  needs:
    - deploy-staging
  environment:
    name: staging
    action: verify

API contract tests (verify API responses match expected schemas):

contract-test:
  stage: verify
  script:
    - npm run test:contract -- --baseUrl=$CI_ENVIRONMENT_URL
  needs:
    - deploy-staging

Performance baseline check:

performance-check:
  stage: verify
  script:
    - |
      RESPONSE_TIME=$(curl -s -o /dev/null -w "%{time_total}" $CI_ENVIRONMENT_URL)
      echo "Homepage response time: ${RESPONSE_TIME}s"
      
      # Fail if homepage takes more than 2 seconds
      python3 -c "
      import sys
      t = float('$RESPONSE_TIME')
      if t > 2.0:
          print(f'FAIL: Response time {t:.2f}s exceeds 2.0s budget')
          sys.exit(1)
      else:
          print(f'PASS: Response time {t:.2f}s within budget')
      "
  needs:
    - deploy-staging

Deployment Gates and Manual Approval

For production deployments, you typically want a manual approval step after staging validation:

deploy-production:
  stage: deploy
  script:
    - ./deploy.sh production
  environment:
    name: production
    url: https://yourapp.com
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
      when: manual
  needs:
    - smoke-test-staging  # Must pass staging smoke tests first

The when: manual requires someone to click "Run" in the GitLab pipeline view. Combined with needs: [smoke-test-staging], the manual trigger is only available after staging validation passes.

Rollback Testing

A deployment process that can't roll back isn't fully tested. GitLab's environment history enables one-click rollback, but you should verify that your rollback mechanism actually works before you need it in a production incident.

Testing rollback in staging:

test-rollback:
  stage: verify
  script:
    - |
      # Deploy current version
      ./deploy.sh staging $CI_COMMIT_SHA
      
      # Verify it's working
      bash scripts/smoke-test.sh https://staging.yourapp.com
      
      # Get previous deployment SHA
      PREVIOUS_SHA=$(git rev-parse HEAD~1)
      
      # Deploy previous version
      ./deploy.sh staging $PREVIOUS_SHA
      
      # Verify previous version works
      bash scripts/smoke-test.sh https://staging.yourapp.com
      
      # Re-deploy current version
      ./deploy.sh staging $CI_COMMIT_SHA
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
      when: manual

Using GitLab's built-in rollback:

GitLab's Environments page shows deployment history with a "Rollback" button for each previous deployment. Clicking Rollback re-runs the original deploy job with the original commit SHA. This uses your existing deployment scripts, so it tests the actual rollback path rather than a separate rollback procedure.

For GitLab's rollback to work, your deploy script must be idempotent — running it again with an older SHA should produce a working deployment of that version. Test this:

verify-rollback-idempotency:
  stage: verify
  script:
    - |
      # Deploy twice to verify idempotency
      ./deploy.sh staging
      sleep 10
      ./deploy.sh staging  # Should complete successfully
      bash scripts/smoke-test.sh $STAGING_URL
  rules:
    - when: manual

Environment Variables and Secrets by Environment

Different environments need different configurations. GitLab's protected variables system allows secrets to be scoped to specific environments:

In GitLab's Settings > CI/CD > Variables, you can set variables as:

  • Protected: Only available on protected branches/tags
  • Masked: Hidden in job logs
  • Scoped to environment: Only available when deploying to a matching environment
deploy-staging:
  script:
    - echo "Using DATABASE_URL=$DATABASE_URL"  # Staging DB URL (scoped variable)
    - ./deploy.sh staging
  environment:
    name: staging

deploy-production:
  script:
    - echo "Using DATABASE_URL=$DATABASE_URL"  # Production DB URL (different scoped variable)
    - ./deploy.sh production
  environment:
    name: production

Both jobs use a variable called DATABASE_URL, but they receive different values based on their target environment — no conditional logic needed in the pipeline configuration.

Monitoring Environment Health

GitLab integrates with Prometheus-based monitoring, though the deeper integration has evolved over time. For basic deployment health visibility, you can use generic alerting:

deploy-production:
  script:
    - ./deploy.sh production
    - |
      # Wait for deployment to stabilize
      sleep 30
      
      # Check error rate in monitoring
      ERROR_RATE=$(curl -s "https://prometheus.yourinfra.com/api/v1/query?query=sum(rate(http_requests_total{status=~'5..'}[5m]))" | jq '.data.result[0].value[1]')
      
      python3 -c "
      rate = float('$ERROR_RATE')
      if rate > 0.01:
          print(f'WARN: Error rate {rate:.4f} above 1%')
          exit(1)
      print(f'OK: Error rate {rate:.4f}')
      "
  environment:
    name: production

Combining deployment events with monitoring alerts creates a complete production quality loop: deploy, verify functionality, verify performance and error rates, confirm or roll back.

Read more

Start now free