Testing

Falco: Runtime Threat Detection Testing for Kubernetes Workloads

HelpMeTest

HelpMeTest

— 9 min read
Falco: Runtime Threat Detection Testing for Kubernetes Workloads

Runtime security is the last line of defense. If a container escapes, an attacker pivots laterally, or malware executes inside your cluster — Falco is what catches it. But Falco rules that go untested are rules that fail silently when it matters most.

This guide covers how to write, validate, and continuously test Falco rules for Kubernetes workloads.

What Falco Does

Falco is a CNCF runtime security tool that monitors Linux system calls (syscalls) using eBPF or kernel modules. It evaluates those syscalls against a rule set and fires alerts when suspicious behavior occurs.

Falco watches for:

  • Shell execution inside containers (execve for /bin/bash, /bin/sh)
  • Sensitive file reads (/etc/shadow, /proc/*/mem)
  • Network connections from unexpected processes
  • Privilege escalation attempts
  • Container namespace escapes

The problem is that rules need tuning. A rule that fires too broadly creates alert fatigue. A rule that's too narrow misses real attacks. Testing is how you find that balance.

Falco Rule Anatomy

A Falco rule has four components:

- rule: Shell Spawned in Container
  desc: A shell was spawned inside a container
  condition: >
    spawned_process and
    container and
    shell_procs and
    not proc.pname in (shell_spawning_binaries)
  output: >
    Shell spawned in a container
    (user=%user.name user_loginuid=%user.loginuid
     container_id=%container.id
     image=%container.image.repository:%container.image.tag
     shell=%proc.name parent=%proc.pname
     cmdline=%proc.cmdline)
  priority: WARNING
  tags: [container, shell, mitre_execution]

Key fields:

  • condition: Falco filter expression using fields from the event context
  • output: What gets logged when the rule fires — include enough context to investigate
  • priority: DEBUG, INFO, WARNING, ERROR, CRITICAL
  • tags: Used for filtering and categorization

Setting Up a Test Environment

Install Falco

# Helm install (recommended for Kubernetes)
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

helm install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  --<span class="hljs-built_in">set driver.kind=modern_ebpf \
  --<span class="hljs-built_in">set falcosidekick.enabled=<span class="hljs-literal">true \
  --<span class="hljs-built_in">set falcosidekick.webui.enabled=<span class="hljs-literal">true

Verify Falco is running:

kubectl get pods -n falco
kubectl logs -n falco -l app.kubernetes.io/name=falco --tail=20

Install falco-event-generator

The falco-event-generator tool triggers synthetic syscall events to validate your rules fire correctly:

kubectl apply -f https://raw.githubusercontent.com/falcosecurity/event-generator/main/deploy/k8s-with-clusterrole.yaml

Or run it as a one-shot job:

kubectl run event-generator \
  --image=falcosecurity/event-generator \
  --restart=Never \
  -- run syscall --loop

Testing Built-In Rules

Validate Default Rules Fire

Run the event generator against the default Falco ruleset and verify alerts appear in Falco logs:

# Trigger all syscall events
kubectl run event-gen \
  --image=falcosecurity/event-generator \
  --restart=Never \
  -- run syscall

<span class="hljs-comment"># Watch Falco logs for alerts
kubectl logs -n falco -l app.kubernetes.io/name=falco -f <span class="hljs-pipe">| grep -E <span class="hljs-string">"Warning|Error|Critical"

Expected output for "shell in container" rule:

Warning Spawned Shell in Container evt_type=execve user=root user_uid=0 user_loginuid=-1
  container_id=abc123 image=falcosecurity/event-generator:latest
  shell=sh parent=event-generator cmdline=sh -c uptime

Check Which Rules Fire for Each Category

# Test only k8s-audit related events
kubectl run event-gen \
  --image=falcosecurity/event-generator \
  --restart=Never \
  -- run k8saudit

<span class="hljs-comment"># Test only syscall events  
kubectl run event-gen \
  --image=falcosecurity/event-generator \
  --restart=Never \
  -- run syscall

Writing Custom Rules

Example: Detect Crypto Mining

Crypto miners make outbound connections to mining pools on known ports and spawn intensive CPU processes. Here's a rule to catch it:

- list: crypto_mining_ports
  items: [3333, 4444, 8333, 8888, 14444, 14433, 45560, 45700]

- list: crypto_mining_binaries
  items: [xmrig, minerd, cpuminer, cgminer, bfgminer, ethminer]

- rule: Crypto Mining Activity Detected
  desc: Process or network behavior consistent with crypto mining
  condition: >
    (spawned_process and proc.name in (crypto_mining_binaries)) or
    (outbound and fd.sport in (crypto_mining_ports))
  output: >
    Crypto mining detected
    (user=%user.name proc=%proc.name cmdline=%proc.cmdline
     container=%container.id image=%container.image.repository
     dest=%fd.rip:%fd.sport)
  priority: CRITICAL
  tags: [container, network, cryptomining, mitre_impact]

Example: Detect Sensitive File Access

- macro: sensitive_files
  condition: >
    fd.name in (/etc/shadow, /etc/sudoers, /root/.ssh/authorized_keys,
                /root/.ssh/id_rsa, /etc/kubernetes/admin.conf)

- rule: Sensitive File Read
  desc: A process read a sensitive file
  condition: >
    open_read and sensitive_files and
    not proc.name in (sshd, sudo, passwd, chage)
  output: >
    Sensitive file opened for reading
    (user=%user.name proc=%proc.name file=%fd.name
     container=%container.id)
  priority: ERROR
  tags: [filesystem, sensitive, mitre_credential_access]

Testing Custom Rules with falco-unit-test

Falco has a built-in unit testing mechanism. Write .yaml test files that describe expected rule behavior:

# tests/shell-in-container.yaml
- rule: Shell Spawned in Container
  tests:
    - test_name: shell_spawned_in_container
      events:
        - type: execve
          fields:
            proc.name: bash
            proc.pname: python3
            container.id: abc123
            user.name: root
      expect_alert: true

    - test_name: shell_spawned_outside_container
      events:
        - type: execve
          fields:
            proc.name: bash
            proc.pname: sshd
            container.id: ""  # Not in a container
            user.name: root
      expect_alert: false

Run unit tests:

falco --test -r custom-rules.yaml tests/shell-in-container.yaml

Integration Testing with a Real Cluster

Test Framework: Bash + kubectl

#!/bin/bash
<span class="hljs-comment"># test-falco-rules.sh

FALCO_NS=<span class="hljs-string">"falco"
ALERT_LOG=<span class="hljs-string">"/tmp/falco-alerts.log"
PASS=0
FAIL=0

<span class="hljs-function">start_collecting_alerts() {
  kubectl logs -n <span class="hljs-string">"$FALCO_NS" -l app.kubernetes.io/name=falco \
    --since=10s -f > <span class="hljs-string">"$ALERT_LOG" &
  COLLECTOR_PID=$!
  <span class="hljs-built_in">sleep 2  <span class="hljs-comment"># Let collector start
}

<span class="hljs-function">stop_collecting_alerts() {
  <span class="hljs-built_in">kill <span class="hljs-variable">$COLLECTOR_PID 2>/dev/null
}

<span class="hljs-function">assert_alert_fired() {
  <span class="hljs-built_in">local rule_name=<span class="hljs-string">"$1"
  <span class="hljs-built_in">local <span class="hljs-built_in">timeout=<span class="hljs-string">"${2:-15}"
  <span class="hljs-built_in">local deadline=$((SECONDS + timeout))
  
  <span class="hljs-keyword">while [ <span class="hljs-variable">$SECONDS -lt <span class="hljs-variable">$deadline ]; <span class="hljs-keyword">do
    <span class="hljs-keyword">if grep -q <span class="hljs-string">"$rule_name" <span class="hljs-string">"$ALERT_LOG"; <span class="hljs-keyword">then
      <span class="hljs-built_in">echo <span class="hljs-string">"PASS: Alert '$rule_name' fired"
      PASS=$((PASS + <span class="hljs-number">1))
      <span class="hljs-built_in">return 0
    <span class="hljs-keyword">fi
    <span class="hljs-built_in">sleep 1
  <span class="hljs-keyword">done
  
  <span class="hljs-built_in">echo <span class="hljs-string">"FAIL: Alert '$rule_name' did not fire within <span class="hljs-variable">${timeout}s"
  FAIL=$((FAIL + <span class="hljs-number">1))
  <span class="hljs-built_in">return 1
}

<span class="hljs-comment"># Test 1: Shell spawned in container
<span class="hljs-function">test_shell_in_container() {
  <span class="hljs-built_in">echo <span class="hljs-string">"--- Test: Shell in Container ---"
  start_collecting_alerts
  
  kubectl run shell-test \
    --image=alpine:latest \
    --restart=Never \
    --<span class="hljs-built_in">rm \
    -- sh -c <span class="hljs-string">"echo test" 2>/dev/null
  
  <span class="hljs-built_in">sleep 5
  stop_collecting_alerts
  assert_alert_fired <span class="hljs-string">"Shell Spawned"
}

<span class="hljs-comment"># Test 2: Sensitive file read
<span class="hljs-function">test_sensitive_file_read() {
  <span class="hljs-built_in">echo <span class="hljs-string">"--- Test: Sensitive File Read ---"
  start_collecting_alerts
  
  kubectl run file-test \
    --image=alpine:latest \
    --restart=Never \
    --<span class="hljs-built_in">rm \
    -- <span class="hljs-built_in">cat /etc/shadow 2>/dev/null
  
  <span class="hljs-built_in">sleep 5
  stop_collecting_alerts
  assert_alert_fired <span class="hljs-string">"Sensitive File"
}

test_shell_in_container
test_sensitive_file_read

<span class="hljs-built_in">echo <span class="hljs-string">""
<span class="hljs-built_in">echo <span class="hljs-string">"Results: $PASS passed, <span class="hljs-variable">$FAIL failed"
[ <span class="hljs-variable">$FAIL -eq 0 ]

Python Test Framework

For larger test suites, use Python with the Kubernetes client:

import time
import subprocess
import pytest
from kubernetes import client, config

config.load_kube_config()
v1 = client.CoreV1Api()

FALCO_NAMESPACE = "falco"
FALCO_LABEL = "app.kubernetes.io/name=falco"


def get_recent_falco_alerts(since_seconds=30):
    """Fetch recent Falco log output."""
    result = subprocess.run(
        ["kubectl", "logs", "-n", FALCO_NAMESPACE,
         "-l", FALCO_LABEL, f"--since={since_seconds}s"],
        capture_output=True, text=True
    )
    return result.stdout


def trigger_shell_in_container():
    """Run a pod that spawns a shell."""
    pod = client.V1Pod(
        metadata=client.V1ObjectMeta(
            name="falco-test-shell",
            namespace="default"
        ),
        spec=client.V1PodSpec(
            restart_policy="Never",
            containers=[client.V1Container(
                name="test",
                image="alpine:latest",
                command=["sh", "-c", "echo triggered && sleep 2"]
            )]
        )
    )
    v1.create_namespaced_pod("default", pod)
    time.sleep(10)
    try:
        v1.delete_namespaced_pod("falco-test-shell", "default")
    except Exception:
        pass


@pytest.fixture(autouse=True)
def clean_test_pods():
    yield
    for name in ["falco-test-shell", "falco-test-file"]:
        try:
            v1.delete_namespaced_pod(name, "default")
        except Exception:
            pass


def test_shell_spawned_in_container_fires_alert():
    trigger_shell_in_container()
    alerts = get_recent_falco_alerts(since_seconds=30)
    assert "Shell" in alerts or "shell" in alerts, \
        f"Expected shell alert in Falco logs. Got:\n{alerts}"


def test_sensitive_file_read_fires_alert():
    pod = client.V1Pod(
        metadata=client.V1ObjectMeta(
            name="falco-test-file",
            namespace="default"
        ),
        spec=client.V1PodSpec(
            restart_policy="Never",
            containers=[client.V1Container(
                name="test",
                image="alpine:latest",
                command=["sh", "-c", "cat /etc/shadow || true"]
            )]
        )
    )
    v1.create_namespaced_pod("default", pod)
    time.sleep(10)
    
    alerts = get_recent_falco_alerts(since_seconds=30)
    assert "shadow" in alerts.lower() or "Sensitive" in alerts, \
        f"Expected sensitive file alert. Got:\n{alerts}"

Run:

pytest tests/test_falco_rules.py -v

CI/CD Integration

GitHub Actions Pipeline

name: Falco Rules CI

on:
  push:
    paths:
      - 'falco-rules/**'
      - 'tests/falco/**'

jobs:
  test-falco-rules:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Falco (no driver mode for CI)
        run: |
          curl -s https://falco.org/repo/falcosecurity-packages.asc | sudo apt-key add -
          echo "deb https://download.falco.org/packages/deb stable main" \
            | sudo tee /etc/apt/sources.list.d/falcosecurity.list
          sudo apt-get update -q
          sudo apt-get install -y falco

      - name: Validate rule syntax
        run: |
          falco -V -r falco-rules/custom-rules.yaml
          echo "Rule syntax valid"

      - name: Run unit tests
        run: |
          for test_file in tests/falco/*.yaml; do
            echo "Running: $test_file"
            falco --test -r falco-rules/custom-rules.yaml "$test_file"
          done

      - name: Check for duplicate rule names
        run: |
          python3 scripts/check-rule-duplicates.py falco-rules/

Alerting Rule Validation Script

#!/usr/bin/env python3
# scripts/check-rule-duplicates.py
import sys
import yaml
from pathlib import Path


def load_rules(directory):
    rules = []
    for path in Path(directory).glob("*.yaml"):
        with open(path) as f:
            docs = yaml.safe_load_all(f)
            for doc in docs:
                if isinstance(doc, list):
                    rules.extend(
                        item for item in doc
                        if isinstance(item, dict) and "rule" in item
                    )
    return rules


def check_duplicates(rules):
    names = [r["rule"] for r in rules]
    seen = set()
    duplicates = []
    for name in names:
        if name in seen:
            duplicates.append(name)
        seen.add(name)
    return duplicates


if __name__ == "__main__":
    directory = sys.argv[1] if len(sys.argv) > 1 else "."
    rules = load_rules(directory)
    duplicates = check_duplicates(rules)
    
    if duplicates:
        print(f"ERROR: Duplicate rule names found: {duplicates}")
        sys.exit(1)
    
    print(f"OK: {len(rules)} rules, no duplicates")

Tuning Rules: Reducing False Positives

False positives are the enemy of runtime security. A rule that fires constantly gets ignored or disabled.

Macros for Exceptions

- macro: trusted_shell_users
  condition: user.name in (root, nobody) and proc.pname in (sshd, sudo)

- macro: maintenance_window
  condition: >
    (evt.time.s >= 0200 and evt.time.s <= 0400)  # 2-4 AM UTC

- rule: Shell Spawned in Container
  condition: >
    spawned_process and container and shell_procs and
    not trusted_shell_users and
    not proc.pname in (shell_spawning_binaries)

Listing Known-Good Containers

- list: trusted_debug_images
  items: [
    "company.registry.io/debug-tools",
    "gcr.io/cloud-builders/kubectl"
  ]

- macro: trusted_container
  condition: container.image.repository in (trusted_debug_images)

- rule: Shell Spawned in Container
  condition: >
    spawned_process and container and shell_procs and
    not trusted_container and
    not proc.pname in (shell_spawning_binaries)

Common Testing Mistakes

Testing in the wrong namespace: Falco watches all namespaces by default, but your test pod and Falco must both be running and reachable. Verify with kubectl get pods -A.

Timing issues: Falco processes syscalls asynchronously. Always add a sleep after triggering an event before checking logs.

Driver mismatch: If Falco uses kmod but your CI only supports modern_ebpf, rules that depend on specific syscall tracing may behave differently. Test with the same driver you use in production.

Rule syntax errors silently skipped: Always validate syntax with falco -V -r your-rules.yaml before running integration tests.

Connecting to HelpMeTest

Once you have Falco alerts flowing to a webhook or log aggregator, you can build automated test scenarios in HelpMeTest that:

  1. Trigger a suspicious action (spawn a shell, read /etc/shadow)
  2. Check your alerting pipeline received and forwarded the Falco event
  3. Verify your on-call rotation got notified within the expected SLA

This closes the loop: not just "does Falco detect it?" but "does our full detection-to-alert pipeline work end to end?"

Summary

Testing Falco rules requires:

  1. Syntax validation — falco -V -r rules.yaml catches YAML and condition errors before deployment
  2. Unit tests — Falco's built-in --test mode validates individual rule logic with synthetic events
  3. Integration tests — Trigger real syscalls in a real cluster and assert alerts appear
  4. CI enforcement — Validate on every rule change, fail the pipeline if tests break
  5. Continuous tuning — Track false positive rates and adjust macros/lists accordingly

Runtime security rules are living documents. The threat landscape changes, your workloads change, and your rules need to keep pace. Automated testing is what makes that evolution safe.

Read more