Testing

Tetragon Security Policy Testing: Testing Cilium Tetragon Security Policies

HelpMeTest

HelpMeTest

— 7 min read
Tetragon Security Policy Testing: Testing Cilium Tetragon Security Policies

Cilium Tetragon brings eBPF-powered runtime security enforcement to Kubernetes. It can block file access, kill processes that violate policy, and generate structured security events at the kernel level — all without modifying your application. But like any security control, Tetragon policies are only as good as the testing behind them. An untested security policy is a liability: you do not know if it actually blocks what you think it blocks, and you do not know if a future Tetragon upgrade will change the behavior.

This post covers how to test Tetragon TracingPolicy resources systematically — from unit-level policy validation through integration tests that verify enforcement in a live cluster.

Understanding Tetragon's Testing Surface

Tetragon has three layers that need testing:

  1. Policy syntax — does the TracingPolicy YAML parse and load without errors?
  2. Event generation — does the policy generate events when the matching condition occurs?
  3. Enforcement — does the policy's action (SIGKILL, override, etc.) actually block the behavior?

Most teams test only layer 1 (it loads, CI is green). Layers 2 and 3 are where real security assurance lives.

The TracingPolicy Under Test

Throughout this post we will test a policy that blocks file writes to /etc/passwd from any process except login:

# policies/block-passwd-write.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: block-passwd-write
spec:
  kprobes:
  - call: "security_file_permission"
    syscall: false
    args:
    - index: 0
      type: "file"
    - index: 1
      type: "int"
    selectors:
    - matchArgs:
      - index: 0
        operator: "Prefix"
        values:
        - "/etc/passwd"
      - index: 1
        operator: "Equal"
        values:
        - "2"   # MAY_WRITE
      matchBinaries:
      - operator: "NotIn"
        values:
        - "/bin/login"
      matchActions:
      - action: Sigkill

Layer 1: Policy Validation

Before applying a policy to any cluster, validate its schema. Tetragon ships a JSON schema for TracingPolicy; use it in CI:

#!/bin/bash
# scripts/validate-policies.sh

SCHEMA_URL="https://raw.githubusercontent.com/cilium/tetragon/main/vendor/github.com/cilium/cilium/pkg/k8s/apis/cilium.io/v1alpha1/crds/cilium.io_tracingpolicies.yaml"

# Install kubeconform if not present
which kubeconform || go install github.com/yannh/kubeconform/cmd/kubeconform@latest

for policy_file in policies/*.yaml; do
    echo "Validating: $policy_file"
    kubeconform \
        -schema-location "$SCHEMA_URL" \
        -strict \
        "$policy_file"

    if [ $? -ne 0 ]; then
        echo "FAIL: $policy_file"
        exit 1
    fi
    echo "PASS: $policy_file"
done

Also validate with kubectl --dry-run:

kubectl apply --dry-run=client -f policies/block-passwd-write.yaml

This catches syntax errors without touching a cluster.

Layer 2: Event Generation Testing

Event testing verifies that Tetragon generates the expected JSON events when a matching operation occurs. Use tetra (the Tetragon CLI) to stream events during a test:

Test Script Pattern

#!/bin/bash
# tests/test-passwd-write-event.sh
set -e

NAMESPACE="kube-system"
EVENT_TIMEOUT=10

echo "=== Test: write to /etc/passwd generates Tetragon event ==="

# Apply the policy
kubectl apply -f policies/block-passwd-write.yaml
sleep 2  # Wait for policy propagation

# Start event streaming in background
EVENTS_FILE=$(mktemp)
kubectl exec -n "$NAMESPACE" ds/tetragon -- \
    tetra getevents \
    --output json \
    --filter-by-policy block-passwd-write \
    > "$EVENTS_FILE" &
STREAM_PID=$!

# Trigger the matching condition in a test pod
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: policy-test-writer
spec:
  containers:
  - name: writer
    image: busybox
    command: ["sh", "-c", "echo test >> /etc/passwd; sleep 5"]
  restartPolicy: Never
EOF

# Wait for the pod to attempt the write
sleep "$EVENT_TIMEOUT"

# Stop event streaming
kill "$STREAM_PID" 2>/dev/null || true

# Clean up test pod
kubectl delete pod policy-test-writer --ignore-not-found

# Assert: at least one event was generated
EVENT_COUNT=$(grep -c '"type":"KPROBE"' "$EVENTS_FILE" 2>/dev/null || echo 0)
rm -f "$EVENTS_FILE"

if [ "$EVENT_COUNT" -gt 0 ]; then
    echo "PASS: $EVENT_COUNT event(s) generated for /etc/passwd write attempt"
else
    echo "FAIL: No events generated — policy may not be loaded correctly"
    kubectl delete -f policies/block-passwd-write.yaml
    exit 1
fi

kubectl delete -f policies/block-passwd-write.yaml

Parsing Event Fields

Tetragon events are structured JSON. Assert on specific fields to confirm the event content matches expectations:

#!/usr/bin/env python3
# tests/validate_event.py
import json, sys

def validate_tetragon_event(event_json_str):
    """Validate a Tetragon KPROBE event matches expectations."""
    event = json.loads(event_json_str)

    assertions = [
        ("type check", event.get("type") == "KPROBE"),
        ("function check", event.get("process_kprobe", {}).get("function_name") == "security_file_permission"),
        ("file path check", "/etc/passwd" in str(event.get("process_kprobe", {}).get("args", []))),
        ("policy name check", event.get("process_kprobe", {}).get("policy_name") == "block-passwd-write"),
        ("action check", "SIGKILL" in str(event.get("process_kprobe", {}).get("action", ""))),
    ]

    failures = []
    for name, result in assertions:
        if result:
            print(f"  PASS: {name}")
        else:
            print(f"  FAIL: {name}")
            failures.append(name)

    return len(failures) == 0

if __name__ == "__main__":
    for line in sys.stdin:
        line = line.strip()
        if not line:
            continue
        if validate_tetragon_event(line):
            print("Event validation: PASS")
        else:
            print("Event validation: FAIL")
            sys.exit(1)

Layer 3: Enforcement Testing

Enforcement testing verifies that the policy's action (SIGKILL in our case) actually prevents the operation. This is the most critical test — policy without enforcement is just logging.

Enforcement Test Pod

# tests/enforcement-test-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: enforcement-test
  labels:
    app: enforcement-test
spec:
  containers:
  - name: test
    image: ubuntu:22.04
    command:
    - bash
    - -c
    - |
      echo "Starting enforcement test"

      # This should be killed by Tetragon
      echo "testentry" >> /etc/passwd
      EXIT_CODE=$?

      if [ $EXIT_CODE -ne 0 ]; then
        echo "WRITE_BLOCKED=true"
      else
        echo "WRITE_BLOCKED=false"
        echo "FAIL: write to /etc/passwd was not blocked"
        exit 1
      fi

      echo "PASS: enforcement verified"
  restartPolicy: Never

Enforcement Test Runner

#!/bin/bash
# tests/test-enforcement.sh
set -e

echo "=== Test: Tetragon SIGKILL enforcement for /etc/passwd write ==="

kubectl apply -f policies/block-passwd-write.yaml
sleep 3

kubectl apply -f tests/enforcement-test-pod.yaml

# Wait for pod completion (with timeout)
TIMEOUT=60
START=$(date +%s)
while true; do
    PHASE=$(kubectl get pod enforcement-test \
        -o jsonpath='{.status.phase}' 2>/dev/null || echo "Unknown")

    if [ "$PHASE" = "Succeeded" ] || [ "$PHASE" = "Failed" ]; then
        break
    fi

    NOW=$(date +%s)
    if [ $((NOW - START)) -gt $TIMEOUT ]; then
        echo "FAIL: pod did not complete within ${TIMEOUT}s"
        kubectl delete pod enforcement-test --ignore-not-found
        kubectl delete -f policies/block-passwd-write.yaml
        exit 1
    fi
    sleep 2
done

# Get logs and check result
LOGS=$(kubectl logs enforcement-test)
echo "$LOGS"

kubectl delete pod enforcement-test --ignore-not-found
kubectl delete -f policies/block-passwd-write.yaml

if echo "$LOGS" | grep -q "PASS: enforcement verified"; then
    echo "=== ENFORCEMENT TEST PASSED ==="
else
    echo "=== ENFORCEMENT TEST FAILED ==="
    exit 1
fi

Testing Policy Allowlists

The policy allows login to write to /etc/passwd. Test that the allowlist works — a false positive that blocks legitimate processes is as bad as a policy that blocks nothing:

#!/bin/bash
# tests/test-allowlist.sh
# Verify that /bin/login is NOT killed when writing to /etc/passwd

echo "=== Test: allowlisted binary is not blocked ==="

kubectl apply -f policies/block-passwd-write.yaml
sleep 3

# Create a pod that simulates login writing to passwd
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: allowlist-test
spec:
  containers:
  - name: test
    image: ubuntu:22.04
    command:
    - bash
    - -c
    - |
      # Copy bash to /bin/login to simulate the allowlisted binary
      # (In a real test, use the actual /bin/login binary)
      cp /bin/bash /tmp/login
      /tmp/login -c "echo allowtest >> /etc/passwd"
      echo "EXIT_CODE=$?"
  restartPolicy: Never
EOF

sleep 15

LOGS=$(kubectl logs allowlist-test 2>/dev/null || echo "")
kubectl delete pod allowlist-test --ignore-not-found
kubectl delete -f policies/block-passwd-write.yaml

if echo "$LOGS" | grep -q "EXIT_CODE=0"; then
    echo "PASS: allowlisted binary was not blocked"
else
    echo "FAIL: allowlisted binary was incorrectly blocked (false positive)"
    exit 1
fi

CI Integration

Bring all three test layers together in a CI pipeline:

name: Tetragon Policy Tests

on:
  push:
    paths:
      - "policies/**"
      - "tests/**"

jobs:
  validate-policies:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - name: Validate policy schemas
        run: bash scripts/validate-policies.sh

  integration-tests:
    needs: validate-policies
    runs-on: self-hosted  # Requires a real k8s cluster
    steps:
      - uses: actions/checkout@v4

      - name: Verify Tetragon is running
        run: |
          kubectl get ds tetragon -n kube-system
          kubectl rollout status ds/tetragon -n kube-system

      - name: Run event generation tests
        run: bash tests/test-passwd-write-event.sh

      - name: Run enforcement tests
        run: bash tests/test-enforcement.sh

      - name: Run allowlist tests
        run: bash tests/test-allowlist.sh

      - name: Run full policy test suite
        run: |
          PASSED=0; FAILED=0
          for test_script in tests/test-*.sh; do
            if bash "$test_script"; then
              PASSED=$((PASSED + 1))
            else
              FAILED=$((FAILED + 1))
            fi
          done
          echo "Results: $PASSED passed, $FAILED failed"
          [ $FAILED -eq 0 ]

Testing Tetragon Upgrades

Tetragon versions change the internal BPF programs and event schema. Before upgrading Tetragon in production, run your policy tests against the new version in a staging cluster:

#!/bin/bash
# scripts/test-upgrade.sh
OLD_VERSION=$1
NEW_VERSION=$2

echo "Testing Tetragon upgrade: $OLD_VERSION -> $NEW_VERSION"

# Install old version
helm upgrade --install tetragon cilium/tetragon \
    --namespace kube-system \
    --version "$OLD_VERSION" \
    --wait

# Baseline test run
bash tests/run-all-tests.sh > results/baseline.txt

# Upgrade
helm upgrade tetragon cilium/tetragon \
    --namespace kube-system \
    --version "$NEW_VERSION" \
    --wait

kubectl rollout status ds/tetragon -n kube-system

# Post-upgrade test run
bash tests/run-all-tests.sh > results/post-upgrade.txt

# Diff results
diff results/baseline.txt results/post-upgrade.txt && \
    echo "PASS: No regressions after upgrade" || \
    echo "FAIL: Test results changed after upgrade — review diff"

End-to-End Security Coverage

Tetragon policy tests verify kernel-level enforcement. They do not verify the rest of your security pipeline: the event ingestion into your SIEM, the alerting rules that fire on those events, or the incident response UI that your security team uses. Testing that full pipeline requires application-level automation.

Teams that use HelpMeTest alongside Tetragon testing cover both layers: Tetragon tests verify that the kernel blocks the right syscalls and generates the right events, while HelpMeTest's AI-powered Robot Framework and Playwright runner exercises the control plane — the dashboards, alert routing, and policy management UI. HelpMeTest's usage-based pricing ($0.003/test run) provides cloud-based test infrastructure without requiring you to stand up and maintain additional tooling.

Policy Test Checklist

Before declaring a Tetragon policy production-ready, verify:

  • Schema validation passes (kubeconform, kubectl --dry-run)
  • Policy loads without errors in the target cluster
  • Matching operations generate events with correct fields
  • Policy action (SIGKILL, override) is enforced — not just logged
  • Allowlisted binaries are not incorrectly blocked (false positive test)
  • Policy survives Tetragon pod restart (state is in kernel, not userspace)
  • Policy behavior is consistent across node restarts
  • Upgrade test passes from current to next Tetragon version

Conclusion

Testing Tetragon security policies is not optional — an untested policy gives you false confidence at best and a silent security gap at worst. The three-layer testing model (validation, event generation, enforcement) covers the full policy lifecycle: syntax correctness, observability output, and actual behavior. Start with schema validation in CI, add event generation tests against a dev cluster, and add enforcement tests before any policy reaches production. Security policies that survive this gauntlet are policies you can trust.

Read more

Start now free