DevOps

Testing Flux GitOps Pipelines: From Kustomization Validation to E2E

HelpMeTest

HelpMeTest

— 15 min read
Testing Flux GitOps Pipelines: From Kustomization Validation to E2E

Flux automates GitOps deployments, which means a broken Kustomization or HelmRelease can silently fail to reconcile — or worse, reconcile incorrectly — without anyone noticing until a production incident. This guide covers the full Flux testing stack: flux build for local manifest rendering, flux diff for live drift detection, envtest for controller testing, and kind-based E2E testing of the full GitOps pipeline.

Key Takeaways

flux build kustomization renders the final manifests without applying them. It resolves patches, overlays, variable substitutions, and Flux-specific transformations. Use it to inspect exactly what Flux would apply, before it does.

flux diff kustomization shows cluster drift. It compares what Flux has applied to the cluster against what the current Git state would produce. Critical for catching configuration drift in live environments.

envtest lets you unit-test Flux controllers. The Flux controllers are standard Kubernetes controllers written with controller-runtime. envtest gives you a real API server without a real cluster — fast, deterministic, and parallelizable.

Test HelmRelease objects separately from their charts. A HelmRelease that references a chart can fail to deploy due to wrong valuesFrom references, incorrect chart version constraints, or missing CRD prerequisites — all independent of the chart's own test coverage.

E2E tests in kind require a real Git server. Flux reconciles from a Git repository. For E2E tests, use Gitea in kind (or GitHub's test repositories) — not mocks. Real reconciliation catches timing bugs and retry logic that mock-based tests miss.

Flux GitOps Architecture and Test Points

A typical Flux setup has three layers, each with distinct failure modes:

Git Repository
    ↓ (GitRepository source)
Flux Source Controller — fetches and caches repo content
    ↓
Kustomization Controller — renders and applies manifests
    ↓ (or)
Helm Controller — manages HelmRelease objects
    ↓
Kubernetes Cluster

Test points:

  1. Source layer — Does the GitRepository successfully authenticate and fetch?
  2. Kustomization layer — Does flux build produce valid manifests? Are patches applied correctly?
  3. HelmRelease layer — Does the chart install and reconcile? Are valuesFrom references valid?
  4. Full E2E — Does a Git commit propagate through to the cluster within the expected time?

flux build kustomization — Rendering Without Applying

flux build kustomization renders the full manifest set that Flux would apply for a given Kustomization:

# Install the Flux CLI
curl -s https://fluxcd.io/install.sh <span class="hljs-pipe">| <span class="hljs-built_in">sudo bash

<span class="hljs-comment"># Build a kustomization (reads from the cluster's GitRepository cache)
flux build kustomization apps \
  --path ./clusters/production/apps \
  --kustomizationfile ./clusters/production/flux-system/kustomization.yaml

<span class="hljs-comment"># Build against a specific branch (useful in PRs)
flux build kustomization apps \
  --path ./clusters/production/apps \
  --revision main@sha1:abc123

<span class="hljs-comment"># Build with variable substitutions applied
flux build kustomization apps \
  --path ./clusters/production/apps \
  --dry-run

Validating Build Output

Pipe the build output through kubeconform to catch manifest errors:

flux build kustomization apps \
  --path ./clusters/production/apps | \
  kubeconform \
    --strict \
    --ignore-missing-schemas \
    --kubernetes-version 1.30.0 \
    --schema-location default \
    --schema-location <span class="hljs-string">'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json'

PR Pipeline with flux build

# .github/workflows/flux-validate.yaml
name: Flux Validation

on:
  pull_request:
    paths:
      - 'clusters/**'
      - 'apps/**'
      - 'infrastructure/**'

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Flux CLI
        run: curl -s https://fluxcd.io/install.sh | sudo bash

      - name: Install kubeconform
        run: |
          curl -Lo kubeconform.tar.gz https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz
          tar xzf kubeconform.tar.gz
          sudo mv kubeconform /usr/local/bin/

      - name: Validate Kustomizations
        run: |
          for path in clusters/*/; do
            echo "Validating: $path"
            flux build kustomization placeholder \
              --path "$path" \
              --dry-run | \
              kubeconform --strict --ignore-missing-schemas
          done

flux diff kustomization — Detecting Drift

flux diff kustomization compares the live cluster state against what Flux would apply from Git. This is the GitOps equivalent of terraform plan against a live environment:

# Show what would change if Flux reconciled right now
flux diff kustomization apps

<span class="hljs-comment"># Diff a specific namespace
flux diff kustomization infrastructure --namespace flux-system

<span class="hljs-comment"># Non-zero exit code if there are differences (useful in CI gating)
flux diff kustomization apps
<span class="hljs-built_in">echo <span class="hljs-string">"Exit code: $?"
<span class="hljs-comment"># 0 = no drift
<span class="hljs-comment"># 1 = drift detected

Automated Drift Detection

Run flux diff on a schedule to alert on drift:

# .github/workflows/drift-detection.yaml
name: Drift Detection

on:
  schedule:
    - cron: '*/30 * * * *'  # every 30 minutes

jobs:
  detect-drift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure kubeconfig
        run: |
          echo "${{ secrets.KUBECONFIG }}" | base64 -d > ~/.kube/config

      - name: Install Flux CLI
        run: curl -s https://fluxcd.io/install.sh | sudo bash

      - name: Check for drift
        id: drift
        run: |
          DRIFT=$(flux diff kustomization apps 2>&1 || true)
          if [ -n "$DRIFT" ]; then
            echo "drift=true" >> $GITHUB_OUTPUT
            echo "drift_output<<EOF" >> $GITHUB_OUTPUT
            echo "$DRIFT" >> $GITHUB_OUTPUT
            echo "EOF" >> $GITHUB_OUTPUT
          fi

      - name: Alert on drift
        if: steps.drift.outputs.drift == 'true'
        run: |
          curl -X POST "${{ secrets.SLACK_WEBHOOK }}" \
            -H 'Content-type: application/json' \
            --data "{\"text\":\"Flux drift detected in production:\\n\`\`\`${{ steps.drift.outputs.drift_output }}\`\`\`\"}"

Testing Kustomizations Locally

Structure for Testable Kustomizations

clusters/
  production/
    flux-system/
      kustomization.yaml
    apps/
      kustomization.yaml
      payments/
        kustomization.yaml
        deployment-patch.yaml
  staging/
    flux-system/
      kustomization.yaml
    apps/
      kustomization.yaml
      payments/
        kustomization.yaml
        deployment-patch.yaml    # staging-specific overrides
base/
  payments/
    deployment.yaml
    service.yaml
    kustomization.yaml

Unit Testing Kustomize Overlays

Test that your overlays produce the expected output without Flux involvement:

#!/usr/bin/env bash
<span class="hljs-comment"># scripts/test-kustomizations.sh

<span class="hljs-built_in">set -e

PASS=0
FAIL=0

<span class="hljs-function">test_kustomization() {
  <span class="hljs-built_in">local name=<span class="hljs-variable">$1
  <span class="hljs-built_in">local path=<span class="hljs-variable">$2
  <span class="hljs-built_in">local expected=<span class="hljs-variable">$3

  actual=$(kustomize build <span class="hljs-string">"$path" 2>&1)

  <span class="hljs-keyword">if [ $? -ne 0 ]; <span class="hljs-keyword">then
    <span class="hljs-built_in">echo <span class="hljs-string">"FAIL: $name — kustomize build failed"
    <span class="hljs-built_in">echo <span class="hljs-string">"$actual"
    FAIL=$((FAIL + <span class="hljs-number">1))
    <span class="hljs-built_in">return
  <span class="hljs-keyword">fi

  <span class="hljs-keyword">if [ -n <span class="hljs-string">"$expected" ]; <span class="hljs-keyword">then
    <span class="hljs-keyword">if diff <(<span class="hljs-built_in">echo <span class="hljs-string">"$actual") <span class="hljs-string">"$expected" > /dev/null 2>&1; <span class="hljs-keyword">then
      <span class="hljs-built_in">echo <span class="hljs-string">"OK: $name matches expected output"
      PASS=$((PASS + <span class="hljs-number">1))
    <span class="hljs-keyword">else
      <span class="hljs-built_in">echo <span class="hljs-string">"FAIL: $name — output differs from expected"
      diff <(<span class="hljs-built_in">echo <span class="hljs-string">"$actual") <span class="hljs-string">"$expected" <span class="hljs-pipe">|| <span class="hljs-literal">true
      FAIL=$((FAIL + <span class="hljs-number">1))
    <span class="hljs-keyword">fi
  <span class="hljs-keyword">else
    <span class="hljs-comment"># Just validate it builds
    <span class="hljs-built_in">echo <span class="hljs-string">"$actual" <span class="hljs-pipe">| kubeconform --strict --ignore-missing-schemas && \
      <span class="hljs-built_in">echo <span class="hljs-string">"OK: $name is valid" && PASS=$((PASS + <span class="hljs-number">1)) <span class="hljs-pipe">|| \
      (<span class="hljs-built_in">echo <span class="hljs-string">"FAIL: $name failed kubeconform validation" && FAIL=$((FAIL + <span class="hljs-number">1)))
  <span class="hljs-keyword">fi
}

test_kustomization <span class="hljs-string">"base/payments" <span class="hljs-string">"base/payments"
test_kustomization <span class="hljs-string">"staging/apps" <span class="hljs-string">"clusters/staging/apps"
test_kustomization <span class="hljs-string">"production/apps" <span class="hljs-string">"clusters/production/apps" <span class="hljs-string">"tests/snapshots/production-apps.yaml"

<span class="hljs-built_in">echo <span class="hljs-string">""
<span class="hljs-built_in">echo <span class="hljs-string">"Kustomization tests: $PASS passed, <span class="hljs-variable">$FAIL failed"
[ <span class="hljs-variable">$FAIL -eq 0 ]

Variable Substitution Testing

Flux Kustomizations support variable substitution from ConfigMaps and Secrets. Test this with a local substitution file:

# base/payments/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
configurations:
  - kustomizeconfig.yaml
# clusters/production/apps/payments/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../../../base/payments
patches:
  - target:
      kind: Deployment
      name: payments
    patch: |
      - op: replace
        path: /spec/replicas
        value: 3
      - op: replace
        path: /spec/template/spec/containers/0/resources/limits/memory
        value: 1Gi

Test that the production overlay produces the expected replica count:

REPLICAS=$(kustomize build clusters/production/apps/payments | \
  yq <span class="hljs-built_in">eval <span class="hljs-string">'select(.kind == "Deployment") | .spec.replicas' -)

<span class="hljs-keyword">if [ <span class="hljs-string">"$REPLICAS" != <span class="hljs-string">"3" ]; <span class="hljs-keyword">then
  <span class="hljs-built_in">echo <span class="hljs-string">"FAIL: expected 3 replicas in production, got $REPLICAS"
  <span class="hljs-built_in">exit 1
<span class="hljs-keyword">fi
<span class="hljs-built_in">echo <span class="hljs-string">"OK: production overlay sets correct replica count"

envtest — Unit Testing Flux Controllers

envtest from sigs.k8s.io/controller-runtime starts a real Kubernetes API server (without a real cluster) for testing controllers. Use it to unit-test custom controllers that integrate with Flux:

// controllers/fluxresource_controller_test.go
package controllers_test

import (
    "context"
    "time"
    "testing"

    . "github.com/onsi/ginkgo/v2"
    . "github.com/onsi/gomega"
    kustomizev1 "github.com/fluxcd/kustomize-controller/api/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/envtest"
)

var (
    testEnv   *envtest.Environment
    k8sClient client.Client
    ctx       context.Context
    cancel    context.CancelFunc
)

var _ = BeforeSuite(func() {
    testEnv = &envtest.Environment{
        CRDDirectoryPaths: []string{
            // Include Flux CRDs for testing
            "vendor/github.com/fluxcd/kustomize-controller/config/crd/bases",
            "vendor/github.com/fluxcd/source-controller/config/crd/bases",
        },
        ErrorIfCRDPathMissing: true,
    }

    cfg, err := testEnv.Start()
    Expect(err).NotTo(HaveOccurred())

    k8sClient, err = client.New(cfg, client.Options{})
    Expect(err).NotTo(HaveOccurred())

    ctx, cancel = context.WithCancel(context.Background())
})

var _ = AfterSuite(func() {
    cancel()
    Expect(testEnv.Stop()).To(Succeed())
})

var _ = Describe("FluxResource Controller", func() {
    It("sets Ready condition when Kustomization reconciles successfully", func() {
        kustomization := &kustomizev1.Kustomization{
            ObjectMeta: metav1.ObjectMeta{
                Name:      "test-app",
                Namespace: "flux-system",
            },
            Spec: kustomizev1.KustomizationSpec{
                Interval: metav1.Duration{Duration: 5 * time.Minute},
                Path:     "./clusters/test/apps",
                SourceRef: kustomizev1.CrossNamespaceSourceReference{
                    Kind: "GitRepository",
                    Name: "flux-system",
                },
            },
        }

        Expect(k8sClient.Create(ctx, kustomization)).To(Succeed())

        // Assert the object was created
        createdKustomization := &kustomizev1.Kustomization{}
        Eventually(func() error {
            return k8sClient.Get(ctx, client.ObjectKeyFromObject(kustomization), createdKustomization)
        }, 10*time.Second, time.Second).Should(Succeed())

        Expect(createdKustomization.Spec.Path).To(Equal("./clusters/test/apps"))
        Expect(createdKustomization.Spec.Interval.Duration).To(Equal(5 * time.Minute))
    })
})

Testing HelmRelease Validation

var _ = Describe("HelmRelease validation", func() {
    It("rejects HelmRelease with invalid chart version constraint", func() {
        helmRelease := &helmv2.HelmRelease{
            ObjectMeta: metav1.ObjectMeta{
                Name:      "bad-release",
                Namespace: "default",
            },
            Spec: helmv2.HelmReleaseSpec{
                Chart: helmv2.HelmChartTemplate{
                    Spec: helmv2.HelmChartTemplateSpec{
                        Chart:   "my-chart",
                        Version: "not-a-semver-constraint!!",  // invalid
                        SourceRef: helmv2.CrossNamespaceObjectReference{
                            Kind: "HelmRepository",
                            Name: "my-repo",
                        },
                    },
                },
            },
        }

        err := k8sClient.Create(ctx, helmRelease)
        Expect(err).To(HaveOccurred())
        Expect(err.Error()).To(ContainSubstring("version"))
    })
})

E2E Testing with Flux in kind

Full E2E tests run Flux in a kind cluster with a real Git repository. Use Gitea for an in-cluster Git server:

Setup: kind + Flux + Gitea

#!/usr/bin/env bash
<span class="hljs-comment"># scripts/e2e-setup.sh

<span class="hljs-built_in">set -e

<span class="hljs-comment"># Create kind cluster
kind create cluster --name flux-e2e

<span class="hljs-comment"># Install Flux
flux install

<span class="hljs-comment"># Install Gitea (in-cluster Git server)
helm repo add gitea-charts https://dl.gitea.io/helm-chart/
helm repo update
helm install gitea gitea-charts/gitea \
  --namespace gitea \
  --create-namespace \
  --<span class="hljs-built_in">set gitea.admin.username=admin \
  --<span class="hljs-built_in">set gitea.admin.password=password \
  --<span class="hljs-built_in">set gitea.admin.email=admin@test.local \
  --<span class="hljs-built_in">set service.http.type=NodePort \
  --<span class="hljs-built_in">wait

<span class="hljs-comment"># Get Gitea's in-cluster URL
GITEA_URL=<span class="hljs-string">"http://gitea-http.gitea.svc.cluster.local:3000"

<span class="hljs-comment"># Create a test repository via Gitea API
kubectl run gitea-init --image=curlimages/curl --<span class="hljs-built_in">rm -it --restart=Never -- \
  curl -X POST <span class="hljs-string">"$GITEA_URL/api/v1/user/repos" \
  -H <span class="hljs-string">"Content-Type: application/json" \
  -u admin:password \
  -d <span class="hljs-string">'{"name":"fleet","private":false,"auto_init":true}'

E2E Test: Full Reconciliation Loop

#!/usr/bin/env bash
<span class="hljs-comment"># scripts/e2e-test.sh

<span class="hljs-built_in">set -e

GITEA_URL=<span class="hljs-string">"http://localhost:3000"  <span class="hljs-comment"># port-forwarded
REPO=<span class="hljs-string">"admin/fleet"

<span class="hljs-comment"># Push test manifests to Gitea
<span class="hljs-built_in">cd /tmp
git <span class="hljs-built_in">clone <span class="hljs-string">"http://admin:password@$GITEA_URL/<span class="hljs-variable">$REPO.git" fleet-test
<span class="hljs-built_in">cd fleet-test

<span class="hljs-built_in">mkdir -p apps/test-app
<span class="hljs-built_in">cat > apps/test-app/deployment.yaml << <span class="hljs-string">'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-app
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-app
  template:
    metadata:
      labels:
        app: test-app
    spec:
      containers:
        - name: app
          image: nginx:1.25
EOF

<span class="hljs-built_in">cat > apps/test-app/kustomization.yaml << <span class="hljs-string">'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
EOF

git add .
git commit -m <span class="hljs-string">"add test-app"
git push origin main

<span class="hljs-comment"># Configure Flux to watch the repository
flux create <span class="hljs-built_in">source git fleet \
  --url=<span class="hljs-string">"http://gitea-http.gitea.svc.cluster.local:3000/$REPO.git" \
  --branch=main \
  --interval=10s

flux create kustomization test-app \
  --<span class="hljs-built_in">source=GitRepository/fleet \
  --path=<span class="hljs-string">"./apps/test-app" \
  --prune=<span class="hljs-literal">true \
  --interval=10s \
  --<span class="hljs-built_in">wait

<span class="hljs-comment"># Assert the deployment was created
kubectl <span class="hljs-built_in">wait --<span class="hljs-keyword">for=condition=available deployment/test-app \
  --namespace=default \
  --<span class="hljs-built_in">timeout=120s

<span class="hljs-built_in">echo <span class="hljs-string">"OK: Initial reconciliation succeeded"

<span class="hljs-comment"># Test update propagation
<span class="hljs-built_in">cd fleet-test
sed -i <span class="hljs-string">'s/replicas: 1/replicas: 2/' apps/test-app/deployment.yaml
git add .
git commit -m <span class="hljs-string">"scale to 2 replicas"
git push origin main

<span class="hljs-comment"># Wait for Flux to detect and apply the change
<span class="hljs-built_in">sleep 20  <span class="hljs-comment"># Flux interval is 10s, give it a full cycle + buffer

REPLICAS=$(kubectl get deployment/test-app -n default -o jsonpath=<span class="hljs-string">'{.spec.replicas}')
<span class="hljs-keyword">if [ <span class="hljs-string">"$REPLICAS" != <span class="hljs-string">"2" ]; <span class="hljs-keyword">then
  <span class="hljs-built_in">echo <span class="hljs-string">"FAIL: Expected 2 replicas after update, got $REPLICAS"
  flux get kustomization test-app
  <span class="hljs-built_in">exit 1
<span class="hljs-keyword">fi

<span class="hljs-built_in">echo <span class="hljs-string">"OK: Update propagated correctly — replicas scaled to 2"

<span class="hljs-comment"># Test prune behavior
<span class="hljs-built_in">cd fleet-test
<span class="hljs-built_in">rm -rf apps/test-app
git add .
git commit -m <span class="hljs-string">"remove test-app"
git push origin main

<span class="hljs-built_in">sleep 20  <span class="hljs-comment"># Wait for prune reconciliation

<span class="hljs-keyword">if kubectl get deployment/test-app -n default &>/dev/null; <span class="hljs-keyword">then
  <span class="hljs-built_in">echo <span class="hljs-string">"FAIL: Deployment still exists after prune"
  <span class="hljs-built_in">exit 1
<span class="hljs-keyword">fi

<span class="hljs-built_in">echo <span class="hljs-string">"OK: Prune removed deployment after Git deletion"
<span class="hljs-built_in">echo <span class="hljs-string">"All E2E tests passed"

Testing HelmRelease Objects

HelmRelease objects have their own failure modes, separate from the chart being deployed:

# Test: valuesFrom references a non-existent ConfigMap
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: payments
  namespace: default
spec:
  interval: 5m
  chart:
    spec:
      chart: payments
      version: ">=1.0.0 <2.0.0"
      sourceRef:
        kind: HelmRepository
        name: platform-charts
  valuesFrom:
    - kind: ConfigMap
      name: payments-config    # this ConfigMap must exist
      valuesKey: values.yaml
    - kind: Secret
      name: payments-secrets   # this Secret must exist
      valuesKey: secret-values.yaml

Test that these references are valid before applying:

#!/usr/bin/env bash
<span class="hljs-comment"># scripts/validate-helmreleases.sh

<span class="hljs-keyword">for file <span class="hljs-keyword">in $(find . -name <span class="hljs-string">"*.yaml" -<span class="hljs-built_in">exec grep -l <span class="hljs-string">"kind: HelmRelease" {} \;); <span class="hljs-keyword">do
  <span class="hljs-built_in">echo <span class="hljs-string">"Validating HelmRelease in $file"

  <span class="hljs-comment"># Extract valuesFrom references
  <span class="hljs-keyword">while IFS= <span class="hljs-built_in">read -r line; <span class="hljs-keyword">do
    KIND=$(<span class="hljs-built_in">echo <span class="hljs-string">"$line" <span class="hljs-pipe">| yq <span class="hljs-built_in">eval <span class="hljs-string">'.kind' -)
    NAME=$(<span class="hljs-built_in">echo <span class="hljs-string">"$line" <span class="hljs-pipe">| yq <span class="hljs-built_in">eval <span class="hljs-string">'.name' -)
    NAMESPACE=$(yq <span class="hljs-built_in">eval <span class="hljs-string">'.metadata.namespace' <span class="hljs-string">"$file")

    <span class="hljs-keyword">if ! kubectl get <span class="hljs-string">"$KIND" <span class="hljs-string">"$NAME" -n <span class="hljs-string">"$NAMESPACE" &>/dev/null; <span class="hljs-keyword">then
      <span class="hljs-built_in">echo <span class="hljs-string">"FAIL: HelmRelease in $file references missing <span class="hljs-variable">$KIND/<span class="hljs-variable">$NAME in namespace <span class="hljs-variable">$NAMESPACE"
      <span class="hljs-built_in">exit 1
    <span class="hljs-keyword">fi
    <span class="hljs-built_in">echo <span class="hljs-string">"OK: $KIND/<span class="hljs-variable">$NAME exists in <span class="hljs-variable">$NAMESPACE"
  <span class="hljs-keyword">done < <(yq <span class="hljs-built_in">eval <span class="hljs-string">'.spec.valuesFrom[]' <span class="hljs-string">"$file")
<span class="hljs-keyword">done

Full CI Pipeline

# .github/workflows/flux-tests.yaml
name: Flux GitOps Tests

on:
  push:
    branches: [main]
  pull_request:
    paths:
      - 'clusters/**'
      - 'apps/**'
      - 'infrastructure/**'

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install tools
        run: |
          curl -s https://fluxcd.io/install.sh | sudo bash
          curl -Lo kubeconform.tar.gz https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz
          tar xzf kubeconform.tar.gz && sudo mv kubeconform /usr/local/bin/
          curl -Lo kustomize.tar.gz https://github.com/kubernetes-sigs/kustomize/releases/latest/download/kustomize_v5.4.1_linux_amd64.tar.gz
          tar xzf kustomize.tar.gz && sudo mv kustomize /usr/local/bin/

      - name: Validate Kustomizations
        run: ./scripts/test-kustomizations.sh

      - name: Validate with flux build
        run: |
          for path in clusters/*/; do
            flux build kustomization placeholder \
              --path "$path" \
              --dry-run | \
              kubeconform --strict --ignore-missing-schemas
          done

  e2e:
    runs-on: ubuntu-latest
    needs: validate
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4

      - name: Set up kind
        uses: helm/kind-action@v1

      - name: Install Flux
        run: |
          curl -s https://fluxcd.io/install.sh | sudo bash
          flux install

      - name: Set up Gitea
        run: |
          helm repo add gitea-charts https://dl.gitea.io/helm-chart/
          helm install gitea gitea-charts/gitea \
            --namespace gitea --create-namespace \
            --set gitea.admin.password=password \
            --wait

      - name: Run E2E tests
        run: ./scripts/e2e-test.sh

Monitoring Flux Reconciliation Health

After deployment, verify Flux is reconciling correctly:

# Check all Flux resources
flux get all --all-namespaces

<span class="hljs-comment"># Check for failed reconciliations
flux get kustomizations --all-namespaces <span class="hljs-pipe">| grep -v <span class="hljs-string">"True"
flux get helmreleases --all-namespaces <span class="hljs-pipe">| grep -v <span class="hljs-string">"True"

<span class="hljs-comment"># Get detailed status for a failing resource
flux describe kustomization apps

<span class="hljs-comment"># Force a reconciliation (bypass interval)
flux reconcile kustomization apps --with-source

<span class="hljs-comment"># Watch reconciliation in real time
flux logs --follow --level=error

Conclusion

Flux GitOps testing is most effective when layered: flux build catches manifest errors before they reach the cluster, flux diff detects drift in live environments, envtest enables fast controller unit testing, and kind-based E2E tests verify the full reconciliation loop including update propagation and prune behavior.

The flux diff command is particularly valuable in production — run it on a schedule and alert on drift to catch out-of-band cluster changes before they cause incidents.

HelpMeTest can monitor your platform engineering pipelines automatically — sign up free

Read more