Kubernetes Testing

ArgoCD GitOps Deployment Validation: Testing Sync, Health Checks, and Progressive Delivery

HelpMeTest

HelpMeTest

— 6 min read
ArgoCD GitOps Deployment Validation: Testing Sync, Health Checks, and Progressive Delivery

ArgoCD manages Kubernetes deployments from Git. When it works, it's invisible — changes merge, ArgoCD syncs, applications update. When it breaks, it can be subtle: a sync that appears healthy but isn't applying changes, a health check that's too permissive, or a rollout that pauses indefinitely without alerting anyone.

Validating ArgoCD deployments means testing more than "did the sync complete?" — it means verifying the deployed application actually works after the sync.

What ArgoCD Deployment Validation Covers

Testing an ArgoCD-managed deployment has several distinct layers:

  1. Manifest validation: Does the Git repository contain valid Kubernetes YAML?
  2. Sync status validation: Did ArgoCD sync successfully? Are all resources in sync?
  3. Health status validation: Are all resources healthy (not just synced)?
  4. Application behavior validation: Does the deployed application work correctly?
  5. Progressive delivery validation: Did canary/blue-green rollouts complete correctly?

Most teams only check layer 2. Layers 3-5 are where production incidents actually come from.

Layer 1: Manifest Validation Before ArgoCD Sees It

Catch manifest errors in CI before they reach ArgoCD:

# .github/workflows/validate-manifests.yml
name: Validate Kubernetes Manifests

on:
  pull_request:
    paths:
      - 'k8s/**'
      - 'charts/**'

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install kubeval
        run: |
          wget https://github.com/instrumenta/kubeval/releases/latest/download/kubeval-linux-amd64.tar.gz
          tar xf kubeval-linux-amd64.tar.gz && sudo mv kubeval /usr/local/bin/

      - name: Validate manifests
        run: |
          find k8s/ -name "*.yaml" -exec kubeval --strict {} \;

      - name: Run kubeconform (faster, more up-to-date schemas)
        uses: docker://ghcr.io/yannh/kubeconform:latest
        with:
          args: "-strict -summary k8s/"

      - name: Run kustomize build (if using kustomize)
        run: |
          kubectl kustomize k8s/overlays/staging | \
          kubeconform -strict -summary -

      - name: Render and validate Helm charts
        run: |
          helm template myapp charts/myapp -f charts/myapp/ci/production-values.yaml | \
          kubeconform -strict -summary -

Layer 2: ArgoCD Sync Status Validation

After a deployment, verify ArgoCD reports everything in sync:

#!/bin/bash
# validate-argocd-sync.sh

APP_NAME=$1
TIMEOUT=300  # 5 minutes

echo "Waiting for ArgoCD app $APP_NAME to sync..."

# Wait for sync to complete
argocd app wait "$APP_NAME" \
  --sync \
  --health \
  --timeout "$TIMEOUT"

if [ $? -ne 0 ]; then
  echo "FAIL: ArgoCD sync timed out or failed"
  argocd app get "$APP_NAME" --refresh
  argocd app history "$APP_NAME"
  exit 1
fi

# Check sync status explicitly
SYNC_STATUS=$(argocd app get "$APP_NAME" -o json | jq -r '.status.sync.status')
HEALTH_STATUS=$(argocd app get "$APP_NAME" -o json | jq -r '.status.health.status')

echo "Sync status: $SYNC_STATUS"
echo "Health status: $HEALTH_STATUS"

if [ "$SYNC_STATUS" != "Synced" ]; then
  echo "FAIL: App is not synced (status: $SYNC_STATUS)"
  exit 1
fi

if [ "$HEALTH_STATUS" != "Healthy" ]; then
  echo "FAIL: App is not healthy (status: $HEALTH_STATUS)"
  argocd app get "$APP_NAME" --refresh
  exit 1
fi

echo "PASS: App is synced and healthy"

Layer 3: Resource Health Validation

ArgoCD's "Healthy" status depends on health checks for each resource type. Understand what these checks actually verify:

  • Deployment: Desired replicas == Ready replicas
  • StatefulSet: Desired replicas == Ready replicas
  • DaemonSet: Desired == Available across all nodes
  • Service: Has endpoints (for ClusterIP/NodePort)
  • Ingress: Has at least one load balancer address (for cloud providers)
  • CronJob: Last schedule time within expected window

A "Healthy" deployment means pods are running — not that they're doing the right thing. Test beyond the health check:

# validate_deployment.py
from kubernetes import client, config
import sys
import time

def validate_deployment_health(namespace: str, deployment_name: str) -> bool:
    config.load_kube_config()
    apps_v1 = client.AppsV1Api()

    dep = apps_v1.read_namespaced_deployment(deployment_name, namespace)

    desired = dep.spec.replicas
    ready = dep.status.ready_replicas or 0

    if ready != desired:
        print(f"FAIL: {desired} desired, {ready} ready")
        return False

    # Check for OOMKilled containers (not visible in ArgoCD health)
    core_v1 = client.CoreV1Api()
    pods = core_v1.list_namespaced_pod(
        namespace,
        label_selector=f"app={deployment_name}"
    )

    for pod in pods.items:
        if pod.status.container_statuses:
            for cs in pod.status.container_statuses:
                if cs.last_state and cs.last_state.terminated:
                    reason = cs.last_state.terminated.reason
                    if reason == "OOMKilled":
                        print(f"WARN: Container {cs.name} was OOMKilled")

    print(f"PASS: {ready}/{desired} replicas ready")
    return True

if __name__ == "__main__":
    success = validate_deployment_health(sys.argv[1], sys.argv[2])
    sys.exit(0 if success else 1)

Layer 4: Application Behavior Validation After Sync

The most important validation: does the application actually work after ArgoCD syncs?

#!/bin/bash
# post-sync-validation.sh
# Called by ArgoCD PostSync hook

SERVICE_URL="https://myapp.staging.example.com"

echo "Running post-sync behavioral validation..."

# Test 1: Health endpoint responds
STATUS=$(curl -s -o /dev/null -w "%{http_code}" --max-time 10 "$SERVICE_URL/health")
if [ "$STATUS" != "200" ]; then
  echo "FAIL: Health endpoint returned $STATUS (expected 200)"
  exit 1
fi
echo "PASS: Health check"

# Test 2: API version matches expected
VERSION=$(curl -s "$SERVICE_URL/version" | jq -r '.version')
EXPECTED_VERSION=$(cat VERSION)
if [ "$VERSION" != "$EXPECTED_VERSION" ]; then
  echo "FAIL: Version mismatch. Got $VERSION, expected $EXPECTED_VERSION"
  exit 1
fi
echo "PASS: Version check ($VERSION)"

# Test 3: Database connection (via app health endpoint)
DB_STATUS=$(curl -s "$SERVICE_URL/health/ready" | jq -r '.database')
if [ "$DB_STATUS" != "ok" ]; then
  echo "FAIL: Database connection is $DB_STATUS"
  exit 1
fi
echo "PASS: Database connection"

echo "All post-sync validations passed."

Register this as an ArgoCD PostSync hook:

# k8s/post-sync-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: post-sync-validation
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  template:
    spec:
      containers:
        - name: validator
          image: curlimages/curl:latest
          command: ["/bin/sh", "/scripts/post-sync-validation.sh"]
          env:
            - name: SERVICE_URL
              valueFrom:
                configMapKeyRef:
                  name: app-config
                  key: service-url
          volumeMounts:
            - name: scripts
              mountPath: /scripts
      volumes:
        - name: scripts
          configMap:
            name: validation-scripts
      restartPolicy: Never

Layer 5: Progressive Delivery Validation With Argo Rollouts

If you use Argo Rollouts for canary or blue-green deployments, validate that the analysis step works:

# k8s/rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: myapp
spec:
  replicas: 4
  strategy:
    canary:
      steps:
        - setWeight: 25  # Send 25% traffic to new version
        - pause: {duration: 30s}
        - analysis:
            templates:
              - templateName: success-rate
        - setWeight: 75
        - pause: {duration: 30s}
        - analysis:
            templates:
              - templateName: success-rate
      canaryService: myapp-canary
      stableService: myapp-stable

---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  metrics:
    - name: success-rate
      interval: 30s
      successCondition: result[0] >= 0.99
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            sum(rate(http_requests_total{job="myapp",status!~"5.."}[2m]))
            /
            sum(rate(http_requests_total{job="myapp"}[2m]))

Validate rollout behavior in CI:

#!/bin/bash
# validate-rollout.sh

ROLLOUT_NAME=$1
TIMEOUT=600  # 10 minutes for full rollout

# Trigger rollout
kubectl argo rollouts set image "$ROLLOUT_NAME" "*=myapp:$NEW_VERSION"

# Watch progress
kubectl argo rollouts status "$ROLLOUT_NAME" --timeout "$TIMEOUT"

if [ $? -ne 0 ]; then
  echo "FAIL: Rollout did not complete"

  # Get analysis results
  kubectl argo rollouts get rollout "$ROLLOUT_NAME" --watch=false

  # Check for aborted rollout
  STATUS=$(kubectl argo rollouts get rollout "$ROLLOUT_NAME" -o json | \
    jq -r '.status.phase')

  if [ "$STATUS" == "Degraded" ]; then
    echo "Rollout was aborted — analysis failed"
    kubectl argo rollouts get rollout "$ROLLOUT_NAME" -o json | \
      jq '.status.currentStepAnalysisRunStatus'
  fi
  exit 1
fi

echo "PASS: Rollout completed successfully"
kubectl argo rollouts get rollout "$ROLLOUT_NAME"

Testing ArgoCD App of Apps

For teams using App of Apps pattern, validate parent and child app sync:

# test_app_of_apps.py
import subprocess
import json
import sys

def get_argocd_apps(parent_app: str) -> list[dict]:
    result = subprocess.run(
        ["argocd", "app", "list", "-o", "json", "--app-namespace", "argocd"],
        capture_output=True, text=True
    )
    apps = json.loads(result.stdout)
    # Filter apps that are children of parent_app
    return [a for a in apps if
            a.get("metadata", {}).get("labels", {}).get("app.kubernetes.io/part-of") == parent_app]

def validate_all_apps_healthy(parent_app: str) -> bool:
    apps = get_argocd_apps(parent_app)
    all_healthy = True

    for app in apps:
        name = app["metadata"]["name"]
        sync_status = app["status"]["sync"]["status"]
        health_status = app["status"]["health"]["status"]

        if sync_status != "Synced" or health_status != "Healthy":
            print(f"FAIL: {name} - sync={sync_status}, health={health_status}")
            all_healthy = False
        else:
            print(f"PASS: {name} - synced and healthy")

    return all_healthy

if __name__ == "__main__":
    parent = sys.argv[1]
    success = validate_all_apps_healthy(parent)
    sys.exit(0 if success else 1)

CI/CD Pipeline Integration

# .github/workflows/argocd-validation.yml
name: ArgoCD Deployment Validation

on:
  push:
    branches: [main]
    paths:
      - 'k8s/**'
      - 'charts/**'

jobs:
  deploy-and-validate:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Install ArgoCD CLI
        run: |
          curl -sSL -o argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
          chmod +x argocd && sudo mv argocd /usr/local/bin/

      - name: Login to ArgoCD
        run: |
          argocd login ${{ secrets.ARGOCD_SERVER }} \
            --username admin \
            --password ${{ secrets.ARGOCD_PASSWORD }} \
            --insecure

      - name: Trigger sync
        run: |
          argocd app sync myapp --prune --force

      - name: Wait for sync and validate
        run: bash scripts/validate-argocd-sync.sh myapp

      - name: Run behavioral tests
        run: |
          /usr/local/bin/await 'curl -sf ${{ vars.APP_URL }}/health'
          pytest tests/smoke/ --base-url=${{ vars.APP_URL }} -v

      - name: Alert on failure
        if: failure()
        run: |
          argocd app get myapp --refresh
          argocd app history myapp

ArgoCD tells you what deployed. Behavioral tests tell you what works. Both are necessary for a reliable GitOps pipeline.

HelpMeTest can run automated behavioral tests as part of your ArgoCD PostSync hooks, giving you continuous verification that every GitOps deployment produces working software. Start free →

Read more

Start now free