DevOps

ArgoCD Testing Strategies: Validate GitOps Deployments Before They Land

HelpMeTest

HelpMeTest

— 13 min read
ArgoCD Testing Strategies: Validate GitOps Deployments Before They Land

ArgoCD makes GitOps deployments feel effortless — until a bad sync silently rolls out a broken config to 40 namespaces via an ApplicationSet. This guide covers how to validate ArgoCD applications, ApplicationSets, and sync policies before they reach a production cluster, using argocd app diff, local kind/k3d clusters, and automated CI checks.

Key Takeaways

argocd app diff is your first line of defense. It shows you exactly what a sync would change — Kubernetes manifests, not Git diffs — before you trigger it. Integrate it into every PR pipeline.

Test ApplicationSets with a local ArgoCD instance. ApplicationSets can generate dozens of applications from a template. A typo in the template propagates to all of them. Test generation logic in a kind cluster before merging.

Dry-run sync policies prevent automated disasters. Setting syncPolicy.automated: {prune: false, selfHeal: false} during testing lets ArgoCD detect drift without automatically fixing it. Use this in staging.

Validate Argo CD Image Updater policies offline. Image Updater's update strategies (semver, digest, latest) can be reasoned about with static policy files before ever running the controller.

Health checks are testable assertions. ArgoCD health status scripts are Lua code. Unit-test them with the argocd-test CLI to catch issues before deploying custom resources.

The Problem With "Just Push and See"

The GitOps promise is that your Git repo is the single source of truth. The hidden danger is that ArgoCD will happily sync whatever ends up in your repo, including broken configurations, wrong image tags, and malformed resource specs.

Common failure patterns:

  • ApplicationSet template bugs — a {{.path.basename}} reference in the wrong field causes all generated applications to get the same name, with the last one winning.
  • Sync policy with prune: true — a deleted file in Git triggers deletion of a running deployment in production.
  • Health status script errors — a custom health check for a CRD throws a Lua error, marking all instances of that resource as Unknown, blocking application sync.
  • Image Updater misconfiguration — a semver constraint that matches pre-release tags starts pulling v2.0.0-rc.1 into production.

All of these are preventable with the right testing strategy.

Local ArgoCD Setup with kind

The fastest feedback loop is a local ArgoCD instance in a kind cluster. This takes about 5 minutes to set up and supports the full ArgoCD feature set:

# Create a kind cluster
kind create cluster --name argocd-test --config - <<<span class="hljs-string">EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
EOF

<span class="hljs-comment"># Install ArgoCD
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

<span class="hljs-comment"># Wait for ArgoCD to be ready
kubectl <span class="hljs-built_in">wait --<span class="hljs-keyword">for=condition=available deployment/argocd-server -n argocd --<span class="hljs-built_in">timeout=120s

<span class="hljs-comment"># Port-forward the API server
kubectl port-forward svc/argocd-server -n argocd 8080:443 &

<span class="hljs-comment"># Get the initial admin password
argocd admin initial-password -n argocd

<span class="hljs-comment"># Log in
argocd login localhost:8080 --username admin --password <password> --insecure

For a faster local loop, use k3d which starts in under 30 seconds:

k3d cluster create argocd-test --agents 1

argocd app diff — Validating Before Syncing

argocd app diff computes the difference between the live cluster state and the desired state from Git without applying anything. It's the equivalent of terraform plan for Kubernetes.

Basic Usage

# Show what would change if you synced right now
argocd app diff my-app

<span class="hljs-comment"># Diff against a specific revision (useful in PR pipelines)
argocd app diff my-app --revision main

<span class="hljs-comment"># Diff against a local directory (bypass Git, test local changes)
argocd app diff my-app --<span class="hljs-built_in">local ./manifests/my-app

Integrating into a PR Pipeline

# .github/workflows/argocd-diff.yaml
name: ArgoCD Diff

on:
  pull_request:
    paths:
      - 'apps/**'
      - 'manifests/**'

jobs:
  diff:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install ArgoCD CLI
        run: |
          curl -sSL -o argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
          chmod +x argocd
          sudo mv argocd /usr/local/bin/

      - name: ArgoCD Login
        run: |
          argocd login ${{ secrets.ARGOCD_SERVER }} \
            --auth-token ${{ secrets.ARGOCD_TOKEN }} \
            --grpc-web

      - name: Compute Diff
        id: diff
        run: |
          DIFF=$(argocd app diff my-app --revision ${{ github.sha }} 2>&1 || true)
          echo "diff<<EOF" >> $GITHUB_OUTPUT
          echo "$DIFF" >> $GITHUB_OUTPUT
          echo "EOF" >> $GITHUB_OUTPUT

      - name: Comment Diff on PR
        uses: actions/github-script@v7
        with:
          script: |
            const diff = `${{ steps.diff.outputs.diff }}`;
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: `## ArgoCD Diff\n\`\`\`diff\n${diff}\n\`\`\``
            });

Exit Codes for Automated Gating

argocd app diff returns exit code 1 when there are differences and 0 when the state is already in sync. In CI, use this to gate merges:

# Fail the build if the app is already out of sync with main
argocd app diff my-app --revision main
<span class="hljs-keyword">if [ $? -eq 1 ]; <span class="hljs-keyword">then
  <span class="hljs-built_in">echo <span class="hljs-string">"WARNING: Cluster is already drifted from main — investigate before merging"
  <span class="hljs-built_in">exit 1
<span class="hljs-keyword">fi

Testing ApplicationSets

ApplicationSets generate ArgoCD Application resources from templates. A single ApplicationSet can control hundreds of applications — making it the highest-leverage place to catch bugs early.

Example ApplicationSet Under Test

# applicationsets/services.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-services
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: https://github.com/myorg/gitops
        revision: HEAD
        directories:
          - path: services/*
  template:
    metadata:
      name: '{{.path.basename}}'
      labels:
        team: '{{.path.segments[1]}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/gitops
        targetRevision: HEAD
        path: '{{.path.path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{.path.basename}}'
      syncPolicy:
        automated:
          prune: false
          selfHeal: false
        syncOptions:
          - CreateNamespace=true

Validating ApplicationSet Output Locally

Apply the ApplicationSet to your local kind cluster and inspect what applications it generates:

# Apply the ApplicationSet
kubectl apply -f applicationsets/services.yaml -n argocd

<span class="hljs-comment"># Wait for the controller to process it
<span class="hljs-built_in">sleep 5

<span class="hljs-comment"># List generated applications
argocd app list

<span class="hljs-comment"># Inspect a specific generated application
argocd app get platform-services-payments --hard-refresh

Automated ApplicationSet Tests

Write a test that applies an ApplicationSet, waits for generation, and asserts the expected applications exist:

#!/usr/bin/env bash
<span class="hljs-comment"># test-appset.sh

<span class="hljs-built_in">set -e

EXPECTED_APPS=(<span class="hljs-string">"payments" <span class="hljs-string">"inventory" <span class="hljs-string">"notifications")

<span class="hljs-comment"># Apply the ApplicationSet
kubectl apply -f applicationsets/services.yaml -n argocd

<span class="hljs-comment"># Wait for applications to appear (max 30s)
<span class="hljs-keyword">for app <span class="hljs-keyword">in <span class="hljs-string">"${EXPECTED_APPS[@]}"; <span class="hljs-keyword">do
  <span class="hljs-built_in">timeout 30 bash -c <span class="hljs-string">"until argocd app get $app &>/dev/null; do sleep 2; done"
  <span class="hljs-built_in">echo <span class="hljs-string">"✓ Application $app generated"
<span class="hljs-keyword">done

<span class="hljs-comment"># Verify application destinations
<span class="hljs-keyword">for app <span class="hljs-keyword">in <span class="hljs-string">"${EXPECTED_APPS[@]}"; <span class="hljs-keyword">do
  NAMESPACE=$(argocd app get <span class="hljs-variable">$app -o json <span class="hljs-pipe">| jq -r <span class="hljs-string">'.spec.destination.namespace')
  <span class="hljs-keyword">if [ <span class="hljs-string">"$NAMESPACE" != <span class="hljs-string">"$app" ]; <span class="hljs-keyword">then
    <span class="hljs-built_in">echo <span class="hljs-string">"✗ Application $app has wrong namespace: <span class="hljs-variable">$NAMESPACE (expected <span class="hljs-variable">$app)"
    <span class="hljs-built_in">exit 1
  <span class="hljs-keyword">fi
  <span class="hljs-built_in">echo <span class="hljs-string">"✓ Application $app targets correct namespace"
<span class="hljs-keyword">done

<span class="hljs-built_in">echo <span class="hljs-string">"All ApplicationSet tests passed"

Testing Sync Policies

Sync policies control when and how ArgoCD applies changes. The most dangerous combinations to test:

Testing prune: true Behavior

prune: true means ArgoCD deletes Kubernetes resources that are no longer in Git. Test this explicitly:

# Step 1: Deploy an app with two resources
<span class="hljs-built_in">cat > /tmp/test-manifests/deployment-a.yaml <<<span class="hljs-string">EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: service-a
spec:
  replicas: 1
  selector:
    matchLabels:
      app: service-a
  template:
    metadata:
      labels:
        app: service-a
    spec:
      containers:
        - name: app
          image: nginx:latest
EOF

<span class="hljs-comment"># Step 2: Apply and sync
argocd app <span class="hljs-built_in">sync prune-test-app --<span class="hljs-built_in">local /tmp/test-manifests

<span class="hljs-comment"># Step 3: Remove one manifest
<span class="hljs-built_in">rm /tmp/test-manifests/deployment-a.yaml

<span class="hljs-comment"># Step 4: Verify prune behavior
argocd app diff prune-test-app --<span class="hljs-built_in">local /tmp/test-manifests
<span class="hljs-comment"># Should show: - Deployment service-a (pruned)

<span class="hljs-comment"># Step 5: Confirm the resource would be deleted (don't actually sync in test)
<span class="hljs-built_in">echo <span class="hljs-string">"Prune test passed — confirmed resource marked for deletion"

Testing selfHeal: true Behavior

selfHeal: true means ArgoCD reverts manual changes made directly to the cluster. Test this in a non-production environment:

# Create a test app with selfHeal enabled
argocd app create selfheal-test \
  --repo https://github.com/myorg/gitops \
  --path apps/test-app \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace selfheal-test \
  --sync-policy automated \
  --self-heal

<span class="hljs-comment"># Manually scale the deployment (simulates unauthorized change)
kubectl scale deployment/test-app -n selfheal-test --replicas=5

<span class="hljs-comment"># Wait for ArgoCD to detect and revert (default check interval: 3m)
<span class="hljs-built_in">sleep 180

<span class="hljs-comment"># Assert replicas were reverted
REPLICAS=$(kubectl get deployment/test-app -n selfheal-test -o jsonpath=<span class="hljs-string">'{.spec.replicas}')
<span class="hljs-keyword">if [ <span class="hljs-string">"$REPLICAS" != <span class="hljs-string">"1" ]; <span class="hljs-keyword">then
  <span class="hljs-built_in">echo <span class="hljs-string">"FAIL: selfHeal did not revert replicas (got $REPLICAS)"
  <span class="hljs-built_in">exit 1
<span class="hljs-keyword">fi
<span class="hljs-built_in">echo <span class="hljs-string">"PASS: selfHeal correctly reverted manual change"

Argo CD Image Updater Testing

Image Updater reads annotations on ArgoCD Application resources and automatically updates image tags. Test your update policies before they run in production.

Policy Annotation Testing

# apps/payments/app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments
  annotations:
    argocd-image-updater.argoproj.io/image-list: app=myorg/payments
    argocd-image-updater.argoproj.io/app.update-strategy: semver
    argocd-image-updater.argoproj.io/app.allow-tags: regexp:^v[0-9]+\.[0-9]+\.[0-9]+$
    argocd-image-updater.argoproj.io/app.ignore-tags: latest,edge

Validating the Update Strategy Offline

Use the argocd-image-updater CLI in dry-run mode to test policy behavior without making changes:

# Install argocd-image-updater CLI
curl -LO https://github.com/argoproj-labs/argocd-image-updater/releases/latest/download/argocd-image-updater_linux_amd64.tar.gz
tar xzf argocd-image-updater_linux_amd64.tar.gz

<span class="hljs-comment"># Test what the updater would do with your current registry state
argocd-image-updater <span class="hljs-built_in">test \
  myorg/payments \
  --update-strategy semver \
  --allow-tags <span class="hljs-string">"regexp:^v[0-9]+\.[0-9]+\.[0-9]+$" \
  --ignore-tags <span class="hljs-string">"latest,edge" \
  --semver-constraint <span class="hljs-string">">=1.0.0 <2.0.0"

This outputs the current latest tag matching your constraints without writing anything.

Testing Health Status Scripts

ArgoCD uses Lua scripts to determine resource health. Custom health checks for CRDs are particularly fragile:

-- argocd-cm custom health check for MyDatabaseCluster
hs = {}
if obj.status ~= nil then
  if obj.status.phase == "Running" then
    hs.status = "Healthy"
    hs.message = "Cluster is running"
  elseif obj.status.phase == "Provisioning" then
    hs.status = "Progressing"
    hs.message = "Cluster is being provisioned"
  else
    hs.status = "Degraded"
    hs.message = "Cluster phase: " .. (obj.status.phase or "unknown")
  end
else
  hs.status = "Progressing"
  hs.message = "Status not yet available"
end
return hs

Test this Lua script using argocd admin settings resource-overrides health:

# Write a test resource to a file
<span class="hljs-built_in">cat > /tmp/test-db-cluster.yaml <<<span class="hljs-string">EOF
apiVersion: db.example.com/v1
kind: MyDatabaseCluster
metadata:
  name: test-cluster
status:
  phase: Running
EOF

<span class="hljs-comment"># Test the health script
argocd admin settings resource-overrides health db.example.com/MyDatabaseCluster \
  /tmp/test-db-cluster.yaml

<span class="hljs-comment"># Expected output:
<span class="hljs-comment"># STATUS: Healthy
<span class="hljs-comment"># MESSAGE: Cluster is running

Automate this across all your custom health scripts:

#!/usr/bin/env bash
<span class="hljs-comment"># test-health-scripts.sh

PASS=0
FAIL=0

<span class="hljs-function">test_health() {
  <span class="hljs-built_in">local resource_file=<span class="hljs-variable">$1
  <span class="hljs-built_in">local expected_status=<span class="hljs-variable">$2
  <span class="hljs-built_in">local group_kind=<span class="hljs-variable">$3

  result=$(argocd admin settings resource-overrides health <span class="hljs-string">"$group_kind" <span class="hljs-string">"$resource_file" 2>&1)
  actual_status=$(<span class="hljs-built_in">echo <span class="hljs-string">"$result" <span class="hljs-pipe">| grep <span class="hljs-string">"^STATUS:" <span class="hljs-pipe">| awk <span class="hljs-string">'{print $2}')

  <span class="hljs-keyword">if [ <span class="hljs-string">"$actual_status" = <span class="hljs-string">"$expected_status" ]; <span class="hljs-keyword">then
    <span class="hljs-built_in">echo <span class="hljs-string">"✓ $group_kind: <span class="hljs-variable">$resource_file → <span class="hljs-variable">$actual_status"
    PASS=$((PASS + <span class="hljs-number">1))
  <span class="hljs-keyword">else
    <span class="hljs-built_in">echo <span class="hljs-string">"✗ $group_kind: <span class="hljs-variable">$resource_file → expected <span class="hljs-variable">$expected_status, got <span class="hljs-variable">$actual_status"
    FAIL=$((FAIL + <span class="hljs-number">1))
  <span class="hljs-keyword">fi
}

test_health fixtures/db-running.yaml Healthy db.example.com/MyDatabaseCluster
test_health fixtures/db-provisioning.yaml Progressing db.example.com/MyDatabaseCluster
test_health fixtures/db-failed.yaml Degraded db.example.com/MyDatabaseCluster
test_health fixtures/db-no-status.yaml Progressing db.example.com/MyDatabaseCluster

<span class="hljs-built_in">echo <span class="hljs-string">""
<span class="hljs-built_in">echo <span class="hljs-string">"Results: $PASS passed, <span class="hljs-variable">$FAIL failed"
[ <span class="hljs-variable">$FAIL -eq 0 ]

App-of-Apps Testing

The app-of-apps pattern uses one ArgoCD application to manage other applications. Test the parent application separately from the child applications:

# apps/root-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/gitops
    targetRevision: HEAD
    path: apps/
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Test the root application's sync in your kind cluster:

# Apply the root app
kubectl apply -f apps/root-app.yaml -n argocd

<span class="hljs-comment"># Trigger a sync and wait for completion
argocd app <span class="hljs-built_in">sync root --<span class="hljs-built_in">timeout 120

<span class="hljs-comment"># Verify all child applications were created
argocd app list <span class="hljs-pipe">| grep -E <span class="hljs-string">"^(payments|inventory|notifications)"

<span class="hljs-comment"># Verify all child applications are healthy
UNHEALTHY=$(argocd app list -o json <span class="hljs-pipe">| jq <span class="hljs-string">'[.[] | select(.status.health.status != "Healthy")] <span class="hljs-pipe">| length')
<span class="hljs-keyword">if [ <span class="hljs-string">"$UNHEALTHY" -gt 0 ]; <span class="hljs-keyword">then
  <span class="hljs-built_in">echo <span class="hljs-string">"FAIL: $UNHEALTHY applications are not Healthy"
  argocd app list -o json <span class="hljs-pipe">| jq <span class="hljs-string">'.[] | select(.status.health.status != "Healthy") <span class="hljs-pipe">| .metadata.name'
  <span class="hljs-built_in">exit 1
<span class="hljs-keyword">fi
<span class="hljs-built_in">echo <span class="hljs-string">"All child applications are Healthy"

Full CI Pipeline

# .github/workflows/argocd-test.yaml
name: ArgoCD Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install tools
        run: |
          # kind
          curl -Lo ./kind https://kind.sigs.k8s.io/dl/latest/kind-linux-amd64
          chmod +x kind && sudo mv kind /usr/local/bin/

          # kubectl
          curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
          chmod +x kubectl && sudo mv kubectl /usr/local/bin/

          # argocd CLI
          curl -sSL -o argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
          chmod +x argocd && sudo mv argocd /usr/local/bin/

      - name: Create kind cluster and install ArgoCD
        run: |
          kind create cluster --name argocd-test
          kubectl create namespace argocd
          kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
          kubectl wait --for=condition=available deployment/argocd-server -n argocd --timeout=120s

      - name: Configure ArgoCD CLI
        run: |
          kubectl port-forward svc/argocd-server -n argocd 8080:443 &
          sleep 5
          PASSWORD=$(argocd admin initial-password -n argocd | head -1)
          argocd login localhost:8080 --username admin --password "$PASSWORD" --insecure

      - name: Test ApplicationSets
        run: ./scripts/test-appset.sh

      - name: Test health scripts
        run: ./scripts/test-health-scripts.sh

      - name: Cleanup
        if: always()
        run: kind delete cluster --name argocd-test

Conclusion

ArgoCD testing is not optional on a platform that manages production workloads. The argocd app diff command alone catches most sync-time surprises; ApplicationSet tests in kind clusters catch template bugs before they propagate; health script unit tests prevent silent degradation of custom resources.

The investment in a local kind-based ArgoCD test environment pays back quickly — most ArgoCD incidents are caused by configurations that would have shown problems clearly in a 10-minute local test run.

HelpMeTest can monitor your platform engineering pipelines automatically — sign up free

Read more