Testing

End-to-End Testing in Kubernetes: A Practical Guide

HelpMeTest

HelpMeTest

— 7 min read
End-to-End Testing in Kubernetes: A Practical Guide

Testing Kubernetes workloads against mocked APIs or unit-tested controllers misses an entire class of bugs — the ones that only appear when your pods, services, ingresses, and RBAC rules interact with a real API server. Running E2E tests against actual clusters in CI used to require expensive dedicated infrastructure. Lightweight Kubernetes distributions changed that. Today you can spin up a full Kubernetes cluster inside a GitHub Actions runner in under two minutes.

Cluster Options for CI

Three tools dominate local and CI Kubernetes testing:

kind (Kubernetes in Docker) runs Kubernetes nodes as Docker containers. It's the most widely adopted choice for CI because it starts quickly, supports multi-node clusters, and integrates well with existing Docker-based pipelines.

k3s is a lightweight Kubernetes distribution from Rancher. It starts faster than kind and uses less memory, making it attractive for resource-constrained CI runners.

minikube is the oldest option, originally designed for local development. It's slower to start and heavier than kind or k3s in CI contexts, but supports a wider range of drivers and addons.

For most teams, kind is the right default.

Setting Up a kind Cluster in CI

# .github/workflows/e2e.yaml
name: E2E Tests
on: [push, pull_request]

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install kind
        run: |
          curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.23.0/kind-linux-amd64
          chmod +x ./kind
          sudo mv ./kind /usr/local/bin/kind

      - name: Create cluster
        run: |
          kind create cluster \
            --name e2e \
            --config kind-config.yaml \
            --wait 2m

      - name: Load images into cluster
        run: |
          docker build -t myapp:e2e-test .
          kind load docker-image myapp:e2e-test --name e2e

      - name: Run E2E tests
        run: ./scripts/run-e2e.sh

      - name: Collect diagnostics on failure
        if: failure()
        run: |
          kubectl get pods -A
          kubectl describe pods -A
          kubectl logs -n myapp --all-containers --previous || true

      - name: Delete cluster
        if: always()
        run: kind delete cluster --name e2e

The kind-config.yaml controls cluster topology:

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/12"

Deploying Test Fixtures

Tests need a known state to run against. Structure your fixtures as plain Kubernetes manifests:

tests/
  fixtures/
    namespace.yaml
    deployment.yaml
    service.yaml
    configmap.yaml
    rbac.yaml
  specs/
    pod-startup.yaml
    service-discovery.yaml
    scaling.yaml

Apply them with a setup script:

#!/usr/bin/env bash
<span class="hljs-comment"># scripts/setup-fixtures.sh
<span class="hljs-built_in">set -euo pipefail

NAMESPACE=<span class="hljs-string">"e2e-tests"

kubectl apply -f tests/fixtures/namespace.yaml
kubectl apply -f tests/fixtures/rbac.yaml
kubectl apply -f tests/fixtures/configmap.yaml
kubectl apply -f tests/fixtures/deployment.yaml
kubectl apply -f tests/fixtures/service.yaml

<span class="hljs-comment"># Wait for deployment to be ready
kubectl rollout status deployment/myapp \
  -n <span class="hljs-string">"$NAMESPACE" \
  --<span class="hljs-built_in">timeout=120s

<span class="hljs-built_in">echo <span class="hljs-string">"Fixtures ready"

The namespace manifest with cleanup annotations:

# tests/fixtures/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: e2e-tests
  labels:
    purpose: e2e-testing
    managed-by: ci

Custom kubectl-Based Test Runner

For teams that want full control without adding new tools, a shell-based test runner works well:

#!/usr/bin/env bash
<span class="hljs-comment"># scripts/run-e2e.sh
<span class="hljs-built_in">set -euo pipefail

NAMESPACE=<span class="hljs-string">"e2e-tests"
FAILED=0

<span class="hljs-function">run_test() {
  <span class="hljs-built_in">local name=<span class="hljs-string">"$1"
  <span class="hljs-built_in">local cmd=<span class="hljs-string">"$2"

  <span class="hljs-built_in">echo -n <span class="hljs-string">"  $name ... "
  <span class="hljs-keyword">if <span class="hljs-built_in">eval <span class="hljs-string">"$cmd" > /tmp/test-output 2>&1; <span class="hljs-keyword">then
    <span class="hljs-built_in">echo <span class="hljs-string">"PASS"
  <span class="hljs-keyword">else
    <span class="hljs-built_in">echo <span class="hljs-string">"FAIL"
    <span class="hljs-built_in">cat /tmp/test-output
    FAILED=$((FAILED + <span class="hljs-number">1))
  <span class="hljs-keyword">fi
}

<span class="hljs-built_in">echo <span class="hljs-string">"=== Setting up fixtures ==="
./scripts/setup-fixtures.sh

<span class="hljs-built_in">echo <span class="hljs-string">""
<span class="hljs-built_in">echo <span class="hljs-string">"=== Running E2E tests ==="

run_test <span class="hljs-string">"deployment is available" \
  <span class="hljs-string">"kubectl get deployment myapp -n $NAMESPACE -o jsonpath='{.status.availableReplicas}' <span class="hljs-pipe">| grep -q '^2$'"

run_test <span class="hljs-string">"service endpoint resolves" \
  <span class="hljs-string">"kubectl run curl-test --image=curlimages/curl --rm -it --restart=Never \
    -n $NAMESPACE -- curl -sf http://myapp:8080/health <span class="hljs-pipe">| grep -q 'ok'"

run_test <span class="hljs-string">"configmap is mounted" \
  <span class="hljs-string">"kubectl exec -n $NAMESPACE deploy/myapp -- \
    cat /etc/config/app.conf <span class="hljs-pipe">| grep -q 'env=test'"

run_test <span class="hljs-string">"hpa scales on load" \
  <span class="hljs-string">"./scripts/trigger-load.sh && sleep 30 && \
    kubectl get hpa myapp -n $NAMESPACE -o jsonpath='{.status.currentReplicas}' <span class="hljs-pipe">| grep -qE '^[3-9]$'"

<span class="hljs-built_in">echo <span class="hljs-string">""
<span class="hljs-built_in">echo <span class="hljs-string">"=== Teardown ==="
kubectl delete namespace <span class="hljs-string">"$NAMESPACE" --<span class="hljs-built_in">wait=<span class="hljs-literal">true

<span class="hljs-keyword">if [ <span class="hljs-string">"$FAILED" -gt 0 ]; <span class="hljs-keyword">then
  <span class="hljs-built_in">echo <span class="hljs-string">"FAILED: $FAILED test(s) failed"
  <span class="hljs-built_in">exit 1
<span class="hljs-keyword">fi

<span class="hljs-built_in">echo <span class="hljs-string">"All tests passed"

KUTTL: Kubernetes Test Tool

KUTTL (kuttl.dev) provides a declarative YAML-based testing framework for Kubernetes. Tests are defined as sequences of steps with assertions:

# tests/kuttl/pod-restart/00-install.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: kuttl-test
data:
  key: value
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: kuttl-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
        - name: app
          image: myapp:e2e-test
          ports:
            - containerPort: 8080
# tests/kuttl/pod-restart/01-assert.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: kuttl-test
status:
  availableReplicas: 1
  readyReplicas: 1

KUTTL compares the cluster state against your assertion files. If the state doesn't match within the timeout, the test fails. Run the suite:

kubectl kuttl test ./tests/kuttl/ \
  --kind-config kind-config.yaml \
  --<span class="hljs-built_in">timeout 120

Chainsaw: Policy-Driven Testing

Chainsaw (from Kyverno) extends the KUTTL model with more expressive assertion syntax, CEL expressions, and better error reporting:

# tests/chainsaw/deployment-health/chainsaw-test.yaml
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
  name: deployment-health
spec:
  steps:
    - name: Deploy application
      try:
        - apply:
            file: deployment.yaml
        - assert:
            file: deployment-ready.yaml
            timeout: 2m

    - name: Verify service responds
      try:
        - script:
            content: |
              kubectl run probe --image=curlimages/curl --rm -it \
                --restart=Never -n $NAMESPACE -- \
                curl -sf http://myapp:8080/health
            timeout: 30s

    - name: Simulate pod failure
      try:
        - delete:
            ref:
              apiVersion: v1
              kind: Pod
              selector:
                matchLabels:
                  app: myapp
        - assert:
            file: deployment-recovered.yaml
            timeout: 60s

Run Chainsaw:

chainsaw test --test-dir ./tests/chainsaw/ \
  --config chainsaw-config.yaml

Chainsaw config:

# chainsaw-config.yaml
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Configuration
metadata:
  name: ci-config
spec:
  timeout: 5m
  skipDelete: false
  template: true
  namespace:
    name: chainsaw-$RANDOM

Namespace Strategy and Teardown

Test isolation requires each test run to operate in its own namespace. Two approaches:

Fixed namespaces with cleanup: Create the namespace at the start of the test run and delete it at the end. Reliable teardown but no parallelism.

Generated namespaces: Create a unique namespace per test or test suite. Allows parallel runs but requires a garbage collection mechanism for leaked namespaces.

# Generate a unique namespace for this CI run
NAMESPACE=<span class="hljs-string">"e2e-$(echo $GITHUB_SHA | head -c 8)-<span class="hljs-subst">$(date +%s)"
<span class="hljs-built_in">export NAMESPACE

kubectl create namespace <span class="hljs-string">"$NAMESPACE"

<span class="hljs-comment"># Always clean up, even on failure
<span class="hljs-built_in">trap <span class="hljs-string">'kubectl delete namespace "$NAMESPACE" --wait=false' EXIT

<span class="hljs-comment"># Run tests
./scripts/run-tests.sh

The --wait=false flag on delete lets your CI job finish quickly — Kubernetes handles the actual resource cleanup asynchronously.

Collecting Diagnostics on Failure

When tests fail in CI, you need enough information to diagnose the problem without re-running locally:

      - name: Collect diagnostics
        if: failure()
        run: |
          echo "=== Pods ==="
          kubectl get pods -A -o wide

          echo "=== Events (last 50) ==="
          kubectl get events -A --sort-by='.lastTimestamp' | tail -50

          echo "=== Application logs ==="
          kubectl logs -n e2e-tests -l app=myapp \
            --all-containers \
            --previous \
            --ignore-errors || true

          echo "=== Resource usage ==="
          kubectl top pods -n e2e-tests || true

      - name: Upload diagnostic bundle
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: e2e-diagnostics-${{ github.run_id }}
          path: /tmp/diagnostics/
          retention-days: 7

Full CI Pipeline

Putting it together:

name: E2E Tests
on:
  push:
    branches: [main]
  pull_request:

jobs:
  e2e:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4

      - name: Set up Go (for controller tools)
        uses: actions/setup-go@v5
        with:
          go-version: '1.22'

      - name: Install tools
        run: |
          # kind
          go install sigs.k8s.io/kind@v0.23.0
          # chainsaw
          go install github.com/kyverno/chainsaw@latest

      - name: Create kind cluster
        run: kind create cluster --name e2e --wait 2m

      - name: Build and load images
        run: |
          docker build -t myapp:e2e .
          kind load docker-image myapp:e2e --name e2e

      - name: Install CRDs and controllers
        run: |
          kubectl apply -f deploy/crds/
          kubectl apply -f deploy/controller/

      - name: Wait for controller
        run: |
          kubectl rollout status deployment/myapp-controller \
            -n myapp-system --timeout=90s

      - name: Run Chainsaw tests
        run: chainsaw test --test-dir ./tests/e2e/

      - name: Collect diagnostics
        if: failure()
        run: |
          kubectl get pods -A
          kubectl describe pods -A
          kubectl logs -n myapp-system --all-containers || true

      - name: Delete cluster
        if: always()
        run: kind delete cluster --name e2e

E2E tests in Kubernetes are slower than unit or integration tests — plan for 5-20 minutes depending on your cluster startup time and test suite size. Run them on push to main branches and on pull requests from trusted contributors, not on every commit to feature branches. The signal-to-noise ratio justifies the wait: these tests catch the category of bugs that only emerge when real Kubernetes scheduling, networking, and RBAC interact with your application.

Read more