DevOps

Testing Flux GitOps Pipelines: From Kustomization Validation to E2E

HelpMeTest

HelpMeTest

— 10 min read
Testing Flux GitOps Pipelines: From Kustomization Validation to E2E

Flux automates GitOps deployments, which means a broken Kustomization or HelmRelease can silently fail to reconcile — or worse, reconcile incorrectly — without anyone noticing until a production incident. This guide covers the full Flux testing stack: flux build for local manifest rendering, flux diff for live drift detection, envtest for controller testing, and kind-based E2E testing of the full GitOps pipeline.

Key Takeaways

flux build kustomization renders the final manifests without applying them. It resolves patches, overlays, variable substitutions, and Flux-specific transformations. Use it to inspect exactly what Flux would apply, before it does.

flux diff kustomization shows cluster drift. It compares what Flux has applied to the cluster against what the current Git state would produce. Critical for catching configuration drift in live environments.

envtest lets you unit-test Flux controllers. The Flux controllers are standard Kubernetes controllers written with controller-runtime. envtest gives you a real API server without a real cluster — fast, deterministic, and parallelizable.

Test HelmRelease objects separately from their charts. A HelmRelease that references a chart can fail to deploy due to wrong valuesFrom references, incorrect chart version constraints, or missing CRD prerequisites — all independent of the chart's own test coverage.

E2E tests in kind require a real Git server. Flux reconciles from a Git repository. For E2E tests, use Gitea in kind (or GitHub's test repositories) — not mocks. Real reconciliation catches timing bugs and retry logic that mock-based tests miss.

Flux GitOps Architecture and Test Points

A typical Flux setup has three layers, each with distinct failure modes:

Git Repository
    ↓ (GitRepository source)
Flux Source Controller — fetches and caches repo content
    ↓
Kustomization Controller — renders and applies manifests
    ↓ (or)
Helm Controller — manages HelmRelease objects
    ↓
Kubernetes Cluster

Test points:

  1. Source layer — Does the GitRepository successfully authenticate and fetch?
  2. Kustomization layer — Does flux build produce valid manifests? Are patches applied correctly?
  3. HelmRelease layer — Does the chart install and reconcile? Are valuesFrom references valid?
  4. Full E2E — Does a Git commit propagate through to the cluster within the expected time?

flux build kustomization — Rendering Without Applying

flux build kustomization renders the full manifest set that Flux would apply for a given Kustomization:

# Install the Flux CLI
curl -s https://fluxcd.io/install.sh | sudo bash

# Build a kustomization (reads from the cluster's GitRepository cache)
flux build kustomization apps \
  --path ./clusters/production/apps \
  --kustomizationfile ./clusters/production/flux-system/kustomization.yaml

# Build against a specific branch (useful in PRs)
flux build kustomization apps \
  --path ./clusters/production/apps \
  --revision main@sha1:abc123

# Build with variable substitutions applied
flux build kustomization apps \
  --path ./clusters/production/apps \
  --dry-run

Validating Build Output

Pipe the build output through kubeconform to catch manifest errors:

flux build kustomization apps \
  --path ./clusters/production/apps | \
  kubeconform \
    --strict \
    --ignore-missing-schemas \
    --kubernetes-version 1.30.0 \
    --schema-location default \
    --schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json'

PR Pipeline with flux build

# .github/workflows/flux-validate.yaml
name: Flux Validation

on:
  pull_request:
    paths:
      - 'clusters/**'
      - 'apps/**'
      - 'infrastructure/**'

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Flux CLI
        run: curl -s https://fluxcd.io/install.sh | sudo bash

      - name: Install kubeconform
        run: |
          curl -Lo kubeconform.tar.gz https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz
          tar xzf kubeconform.tar.gz
          sudo mv kubeconform /usr/local/bin/

      - name: Validate Kustomizations
        run: |
          for path in clusters/*/; do
            echo "Validating: $path"
            flux build kustomization placeholder \
              --path "$path" \
              --dry-run | \
              kubeconform --strict --ignore-missing-schemas
          done

flux diff kustomization — Detecting Drift

flux diff kustomization compares the live cluster state against what Flux would apply from Git. This is the GitOps equivalent of terraform plan against a live environment:

# Show what would change if Flux reconciled right now
flux diff kustomization apps

# Diff a specific namespace
flux diff kustomization infrastructure --namespace flux-system

# Non-zero exit code if there are differences (useful in CI gating)
flux diff kustomization apps
echo "Exit code: $?"
# 0 = no drift
# 1 = drift detected

Automated Drift Detection

Run flux diff on a schedule to alert on drift:

# .github/workflows/drift-detection.yaml
name: Drift Detection

on:
  schedule:
    - cron: '*/30 * * * *'  # every 30 minutes

jobs:
  detect-drift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure kubeconfig
        run: |
          echo "${{ secrets.KUBECONFIG }}" | base64 -d > ~/.kube/config

      - name: Install Flux CLI
        run: curl -s https://fluxcd.io/install.sh | sudo bash

      - name: Check for drift
        id: drift
        run: |
          DRIFT=$(flux diff kustomization apps 2>&1 || true)
          if [ -n "$DRIFT" ]; then
            echo "drift=true" >> $GITHUB_OUTPUT
            echo "drift_output<<EOF" >> $GITHUB_OUTPUT
            echo "$DRIFT" >> $GITHUB_OUTPUT
            echo "EOF" >> $GITHUB_OUTPUT
          fi

      - name: Alert on drift
        if: steps.drift.outputs.drift == 'true'
        run: |
          curl -X POST "${{ secrets.SLACK_WEBHOOK }}" \
            -H 'Content-type: application/json' \
            --data "{\"text\":\"Flux drift detected in production:\\n\`\`\`${{ steps.drift.outputs.drift_output }}\`\`\`\"}"

Testing Kustomizations Locally

Structure for Testable Kustomizations

clusters/
  production/
    flux-system/
      kustomization.yaml
    apps/
      kustomization.yaml
      payments/
        kustomization.yaml
        deployment-patch.yaml
  staging/
    flux-system/
      kustomization.yaml
    apps/
      kustomization.yaml
      payments/
        kustomization.yaml
        deployment-patch.yaml    # staging-specific overrides
base/
  payments/
    deployment.yaml
    service.yaml
    kustomization.yaml

Unit Testing Kustomize Overlays

Test that your overlays produce the expected output without Flux involvement:

#!/usr/bin/env bash
# scripts/test-kustomizations.sh

set -e

PASS=0
FAIL=0

test_kustomization() {
  local name=$1
  local path=$2
  local expected=$3

  actual=$(kustomize build "$path" 2>&1)

  if [ $? -ne 0 ]; then
    echo "FAIL: $name — kustomize build failed"
    echo "$actual"
    FAIL=$((FAIL + 1))
    return
  fi

  if [ -n "$expected" ]; then
    if diff <(echo "$actual") "$expected" > /dev/null 2>&1; then
      echo "OK: $name matches expected output"
      PASS=$((PASS + 1))
    else
      echo "FAIL: $name — output differs from expected"
      diff <(echo "$actual") "$expected" || true
      FAIL=$((FAIL + 1))
    fi
  else
    # Just validate it builds
    echo "$actual" | kubeconform --strict --ignore-missing-schemas && \
      echo "OK: $name is valid" && PASS=$((PASS + 1)) || \
      (echo "FAIL: $name failed kubeconform validation" && FAIL=$((FAIL + 1)))
  fi
}

test_kustomization "base/payments" "base/payments"
test_kustomization "staging/apps" "clusters/staging/apps"
test_kustomization "production/apps" "clusters/production/apps" "tests/snapshots/production-apps.yaml"

echo ""
echo "Kustomization tests: $PASS passed, $FAIL failed"
[ $FAIL -eq 0 ]

Variable Substitution Testing

Flux Kustomizations support variable substitution from ConfigMaps and Secrets. Test this with a local substitution file:

# base/payments/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
configurations:
  - kustomizeconfig.yaml
# clusters/production/apps/payments/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../../../base/payments
patches:
  - target:
      kind: Deployment
      name: payments
    patch: |
      - op: replace
        path: /spec/replicas
        value: 3
      - op: replace
        path: /spec/template/spec/containers/0/resources/limits/memory
        value: 1Gi

Test that the production overlay produces the expected replica count:

REPLICAS=$(kustomize build clusters/production/apps/payments | \
  yq eval 'select(.kind == "Deployment") | .spec.replicas' -)

if [ "$REPLICAS" != "3" ]; then
  echo "FAIL: expected 3 replicas in production, got $REPLICAS"
  exit 1
fi
echo "OK: production overlay sets correct replica count"

envtest — Unit Testing Flux Controllers

envtest from sigs.k8s.io/controller-runtime starts a real Kubernetes API server (without a real cluster) for testing controllers. Use it to unit-test custom controllers that integrate with Flux:

// controllers/fluxresource_controller_test.go
package controllers_test

import (
    "context"
    "time"
    "testing"

    . "github.com/onsi/ginkgo/v2"
    . "github.com/onsi/gomega"
    kustomizev1 "github.com/fluxcd/kustomize-controller/api/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/envtest"
)

var (
    testEnv   *envtest.Environment
    k8sClient client.Client
    ctx       context.Context
    cancel    context.CancelFunc
)

var _ = BeforeSuite(func() {
    testEnv = &envtest.Environment{
        CRDDirectoryPaths: []string{
            // Include Flux CRDs for testing
            "vendor/github.com/fluxcd/kustomize-controller/config/crd/bases",
            "vendor/github.com/fluxcd/source-controller/config/crd/bases",
        },
        ErrorIfCRDPathMissing: true,
    }

    cfg, err := testEnv.Start()
    Expect(err).NotTo(HaveOccurred())

    k8sClient, err = client.New(cfg, client.Options{})
    Expect(err).NotTo(HaveOccurred())

    ctx, cancel = context.WithCancel(context.Background())
})

var _ = AfterSuite(func() {
    cancel()
    Expect(testEnv.Stop()).To(Succeed())
})

var _ = Describe("FluxResource Controller", func() {
    It("sets Ready condition when Kustomization reconciles successfully", func() {
        kustomization := &kustomizev1.Kustomization{
            ObjectMeta: metav1.ObjectMeta{
                Name:      "test-app",
                Namespace: "flux-system",
            },
            Spec: kustomizev1.KustomizationSpec{
                Interval: metav1.Duration{Duration: 5 * time.Minute},
                Path:     "./clusters/test/apps",
                SourceRef: kustomizev1.CrossNamespaceSourceReference{
                    Kind: "GitRepository",
                    Name: "flux-system",
                },
            },
        }

        Expect(k8sClient.Create(ctx, kustomization)).To(Succeed())

        // Assert the object was created
        createdKustomization := &kustomizev1.Kustomization{}
        Eventually(func() error {
            return k8sClient.Get(ctx, client.ObjectKeyFromObject(kustomization), createdKustomization)
        }, 10*time.Second, time.Second).Should(Succeed())

        Expect(createdKustomization.Spec.Path).To(Equal("./clusters/test/apps"))
        Expect(createdKustomization.Spec.Interval.Duration).To(Equal(5 * time.Minute))
    })
})

Testing HelmRelease Validation

var _ = Describe("HelmRelease validation", func() {
    It("rejects HelmRelease with invalid chart version constraint", func() {
        helmRelease := &helmv2.HelmRelease{
            ObjectMeta: metav1.ObjectMeta{
                Name:      "bad-release",
                Namespace: "default",
            },
            Spec: helmv2.HelmReleaseSpec{
                Chart: helmv2.HelmChartTemplate{
                    Spec: helmv2.HelmChartTemplateSpec{
                        Chart:   "my-chart",
                        Version: "not-a-semver-constraint!!",  // invalid
                        SourceRef: helmv2.CrossNamespaceObjectReference{
                            Kind: "HelmRepository",
                            Name: "my-repo",
                        },
                    },
                },
            },
        }

        err := k8sClient.Create(ctx, helmRelease)
        Expect(err).To(HaveOccurred())
        Expect(err.Error()).To(ContainSubstring("version"))
    })
})

E2E Testing with Flux in kind

Full E2E tests run Flux in a kind cluster with a real Git repository. Use Gitea for an in-cluster Git server:

Setup: kind + Flux + Gitea

#!/usr/bin/env bash
# scripts/e2e-setup.sh

set -e

# Create kind cluster
kind create cluster --name flux-e2e

# Install Flux
flux install

# Install Gitea (in-cluster Git server)
helm repo add gitea-charts https://dl.gitea.io/helm-chart/
helm repo update
helm install gitea gitea-charts/gitea \
  --namespace gitea \
  --create-namespace \
  --set gitea.admin.username=admin \
  --set gitea.admin.password=password \
  --set gitea.admin.email=admin@test.local \
  --set service.http.type=NodePort \
  --wait

# Get Gitea's in-cluster URL
GITEA_URL="http://gitea-http.gitea.svc.cluster.local:3000"

# Create a test repository via Gitea API
kubectl run gitea-init --image=curlimages/curl --rm -it --restart=Never -- \
  curl -X POST "$GITEA_URL/api/v1/user/repos" \
  -H "Content-Type: application/json" \
  -u admin:password \
  -d '{"name":"fleet","private":false,"auto_init":true}'

E2E Test: Full Reconciliation Loop

#!/usr/bin/env bash
# scripts/e2e-test.sh

set -e

GITEA_URL="http://localhost:3000"  # port-forwarded
REPO="admin/fleet"

# Push test manifests to Gitea
cd /tmp
git clone "http://admin:password@$GITEA_URL/$REPO.git" fleet-test
cd fleet-test

mkdir -p apps/test-app
cat > apps/test-app/deployment.yaml << 'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-app
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-app
  template:
    metadata:
      labels:
        app: test-app
    spec:
      containers:
        - name: app
          image: nginx:1.25
EOF

cat > apps/test-app/kustomization.yaml << 'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
EOF

git add .
git commit -m "add test-app"
git push origin main

# Configure Flux to watch the repository
flux create source git fleet \
  --url="http://gitea-http.gitea.svc.cluster.local:3000/$REPO.git" \
  --branch=main \
  --interval=10s

flux create kustomization test-app \
  --source=GitRepository/fleet \
  --path="./apps/test-app" \
  --prune=true \
  --interval=10s \
  --wait

# Assert the deployment was created
kubectl wait --for=condition=available deployment/test-app \
  --namespace=default \
  --timeout=120s

echo "OK: Initial reconciliation succeeded"

# Test update propagation
cd fleet-test
sed -i 's/replicas: 1/replicas: 2/' apps/test-app/deployment.yaml
git add .
git commit -m "scale to 2 replicas"
git push origin main

# Wait for Flux to detect and apply the change
sleep 20  # Flux interval is 10s, give it a full cycle + buffer

REPLICAS=$(kubectl get deployment/test-app -n default -o jsonpath='{.spec.replicas}')
if [ "$REPLICAS" != "2" ]; then
  echo "FAIL: Expected 2 replicas after update, got $REPLICAS"
  flux get kustomization test-app
  exit 1
fi

echo "OK: Update propagated correctly — replicas scaled to 2"

# Test prune behavior
cd fleet-test
rm -rf apps/test-app
git add .
git commit -m "remove test-app"
git push origin main

sleep 20  # Wait for prune reconciliation

if kubectl get deployment/test-app -n default &>/dev/null; then
  echo "FAIL: Deployment still exists after prune"
  exit 1
fi

echo "OK: Prune removed deployment after Git deletion"
echo "All E2E tests passed"

Testing HelmRelease Objects

HelmRelease objects have their own failure modes, separate from the chart being deployed:

# Test: valuesFrom references a non-existent ConfigMap
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: payments
  namespace: default
spec:
  interval: 5m
  chart:
    spec:
      chart: payments
      version: ">=1.0.0 <2.0.0"
      sourceRef:
        kind: HelmRepository
        name: platform-charts
  valuesFrom:
    - kind: ConfigMap
      name: payments-config    # this ConfigMap must exist
      valuesKey: values.yaml
    - kind: Secret
      name: payments-secrets   # this Secret must exist
      valuesKey: secret-values.yaml

Test that these references are valid before applying:

#!/usr/bin/env bash
# scripts/validate-helmreleases.sh

for file in $(find . -name "*.yaml" -exec grep -l "kind: HelmRelease" {} \;); do
  echo "Validating HelmRelease in $file"

  # Extract valuesFrom references
  while IFS= read -r line; do
    KIND=$(echo "$line" | yq eval '.kind' -)
    NAME=$(echo "$line" | yq eval '.name' -)
    NAMESPACE=$(yq eval '.metadata.namespace' "$file")

    if ! kubectl get "$KIND" "$NAME" -n "$NAMESPACE" &>/dev/null; then
      echo "FAIL: HelmRelease in $file references missing $KIND/$NAME in namespace $NAMESPACE"
      exit 1
    fi
    echo "OK: $KIND/$NAME exists in $NAMESPACE"
  done < <(yq eval '.spec.valuesFrom[]' "$file")
done

Full CI Pipeline

# .github/workflows/flux-tests.yaml
name: Flux GitOps Tests

on:
  push:
    branches: [main]
  pull_request:
    paths:
      - 'clusters/**'
      - 'apps/**'
      - 'infrastructure/**'

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install tools
        run: |
          curl -s https://fluxcd.io/install.sh | sudo bash
          curl -Lo kubeconform.tar.gz https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz
          tar xzf kubeconform.tar.gz && sudo mv kubeconform /usr/local/bin/
          curl -Lo kustomize.tar.gz https://github.com/kubernetes-sigs/kustomize/releases/latest/download/kustomize_v5.4.1_linux_amd64.tar.gz
          tar xzf kustomize.tar.gz && sudo mv kustomize /usr/local/bin/

      - name: Validate Kustomizations
        run: ./scripts/test-kustomizations.sh

      - name: Validate with flux build
        run: |
          for path in clusters/*/; do
            flux build kustomization placeholder \
              --path "$path" \
              --dry-run | \
              kubeconform --strict --ignore-missing-schemas
          done

  e2e:
    runs-on: ubuntu-latest
    needs: validate
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4

      - name: Set up kind
        uses: helm/kind-action@v1

      - name: Install Flux
        run: |
          curl -s https://fluxcd.io/install.sh | sudo bash
          flux install

      - name: Set up Gitea
        run: |
          helm repo add gitea-charts https://dl.gitea.io/helm-chart/
          helm install gitea gitea-charts/gitea \
            --namespace gitea --create-namespace \
            --set gitea.admin.password=password \
            --wait

      - name: Run E2E tests
        run: ./scripts/e2e-test.sh

Monitoring Flux Reconciliation Health

After deployment, verify Flux is reconciling correctly:

# Check all Flux resources
flux get all --all-namespaces

# Check for failed reconciliations
flux get kustomizations --all-namespaces | grep -v "True"
flux get helmreleases --all-namespaces | grep -v "True"

# Get detailed status for a failing resource
flux describe kustomization apps

# Force a reconciliation (bypass interval)
flux reconcile kustomization apps --with-source

# Watch reconciliation in real time
flux logs --follow --level=error

Conclusion

Flux GitOps testing is most effective when layered: flux build catches manifest errors before they reach the cluster, flux diff detects drift in live environments, envtest enables fast controller unit testing, and kind-based E2E tests verify the full reconciliation loop including update propagation and prune behavior.

The flux diff command is particularly valuable in production — run it on a schedule and alert on drift to catch out-of-band cluster changes before they cause incidents.

HelpMeTest can monitor your platform engineering pipelines automatically — sign up free

Read more

Start now free