Testing

BPF CI Testing Harness: eBPF CI Testing Infrastructure

HelpMeTest

HelpMeTest

— 7 min read
BPF CI Testing Harness: eBPF CI Testing Infrastructure

The Linux kernel's BPF subsystem has one of the most sophisticated CI testing infrastructures in open-source software. The BPF selftests, the netdev CI, the QEMU-based kernel test framework — these systems catch regressions in hundreds of BPF programs across dozens of kernel versions before any code merges. If you maintain eBPF tooling, kernel modules, or BPF-based security products, you need a similar harness. This post shows you how to build one.

Why BPF CI Is Different

BPF programs are compiled into a virtual instruction set, verified by the kernel at load time, and executed in a restricted environment. Testing them requires a real kernel — not a mock, not a container, not a shim. The kernel version matters, the BTF availability matters, the enabled config options matter.

This means BPF CI has fundamentally different requirements from application CI:

  • Kernel isolation — tests must run in VMs with controlled kernel versions, not shared hosts
  • Verifier coverage — the BPF verifier's behavior must be part of what you test
  • Multiple kernel targets — a program valid on 6.1 may fail the verifier on 5.15
  • Privilege requirements — BPF syscalls require CAP_BPF/CAP_PERFMON or root
  • Hardware-specific paths — some BPF programs behave differently on different CPU architectures

Building a harness that handles all of this without becoming a maintenance nightmare requires deliberate design.

The Reference: Linux Kernel BPF Selftests

The Linux kernel ships BPF selftests in tools/testing/selftests/bpf/. Before building your own infrastructure, understand what the upstream selftests provide:

tools/testing/selftests/bpf/
├── prog_tests/           # C-based selftest programs
├── progs/                # BPF C programs under test
├── test_progs.c          # Main test runner
├── test_maps             # Map-specific tests
└── Makefile

The test_progs runner supports filtering, parallelism, and JSON output. If your project extends or uses upstream BPF features, running the relevant upstream selftests in your CI is a baseline requirement.

QEMU-Based Kernel Test VMs

The most reliable BPF CI approach boots a minimal Linux VM in QEMU for each test run. This gives you kernel isolation, reproducibility, and the ability to test against any kernel version.

The vmtest Approach

The BPF community uses a tool called vmtest (part of the drgn project ecosystem) to boot QEMU VMs and run tests. Here is a simplified version you can adapt:

#!/bin/bash
<span class="hljs-comment"># scripts/run-bpf-tests-in-vm.sh

KERNEL_IMAGE=<span class="hljs-variable">${1:-./vmlinuz}
TEST_BINARY=<span class="hljs-variable">${2:-./tests/test_my_prog}
ROOTFS=<span class="hljs-variable">${3:-./rootfs.ext4}

qemu-system-x86_64 \
  -kernel <span class="hljs-string">"$KERNEL_IMAGE" \
  -append <span class="hljs-string">"console=ttyS0 root=/dev/vda rw init=/run-tests.sh" \
  -drive file=<span class="hljs-string">"$ROOTFS",format=raw,<span class="hljs-keyword">if=virtio \
  -virtfs <span class="hljs-built_in">local,path=<span class="hljs-string">"$(pwd)",mount_tag=hostshare,security_model=none \
  -m 2G \
  -smp 4 \
  -nographic \
  -serial mon:stdio \
  -no-reboot

The run-tests.sh init script mounts the host share, runs your test binary, writes results to a known path, then powers off the VM. CI reads the exit code from the QEMU process.

Building a Minimal Rootfs

A test rootfs only needs BusyBox, libbpf, and your test binaries. Build it with:

#!/bin/bash
<span class="hljs-comment"># scripts/build-rootfs.sh
<span class="hljs-built_in">set -e

ROOTFS=<span class="hljs-string">"rootfs.ext4"
MOUNT=<span class="hljs-string">"rootfs-mount"

<span class="hljs-built_in">dd <span class="hljs-keyword">if=/dev/zero of=<span class="hljs-string">"$ROOTFS" bs=1M count=512
mkfs.ext4 -q <span class="hljs-string">"$ROOTFS"
<span class="hljs-built_in">mkdir -p <span class="hljs-string">"$MOUNT"
<span class="hljs-built_in">sudo mount -o loop <span class="hljs-string">"$ROOTFS" <span class="hljs-string">"$MOUNT"

<span class="hljs-comment"># Install BusyBox
<span class="hljs-built_in">sudo <span class="hljs-built_in">cp $(<span class="hljs-built_in">which busybox) <span class="hljs-string">"$MOUNT/bin/"
<span class="hljs-built_in">sudo <span class="hljs-string">"$MOUNT/bin/busybox" --install <span class="hljs-string">"$MOUNT/bin"

<span class="hljs-comment"># Copy test binaries
<span class="hljs-built_in">sudo <span class="hljs-built_in">cp tests/test_my_prog <span class="hljs-string">"$MOUNT/"
<span class="hljs-built_in">sudo <span class="hljs-built_in">cp scripts/run-tests-init.sh <span class="hljs-string">"$MOUNT/run-tests.sh"
<span class="hljs-built_in">sudo <span class="hljs-built_in">chmod +x <span class="hljs-string">"$MOUNT/run-tests.sh"

<span class="hljs-comment"># Install libbpf
<span class="hljs-built_in">sudo <span class="hljs-built_in">cp /usr/lib/x86_64-linux-gnu/libbpf.so.0 <span class="hljs-string">"$MOUNT/lib/"

<span class="hljs-built_in">sudo umount <span class="hljs-string">"$MOUNT"

Kernel Build Configuration

For BPF testing, your kernel config must include:

CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_DEBUG_INFO_BTF=y
CONFIG_BPF_EVENTS=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_STREAM_PARSER=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_HAVE_EBPF_JIT=y

Store this as kernel-configs/bpf-test.config and use it to build test kernels deterministically.

GitHub Actions Workflow

Here is a production-ready GitHub Actions configuration for BPF CI:

name: BPF CI

on:
  push:
    branches: [main]
  pull_request:

jobs:
  build-bpf:
    runs-on: ubuntu-22.04
    outputs:
      artifact-id: ${{ steps.upload.outputs.artifact-id }}
    steps:
      - uses: actions/checkout@v4
        with:
          submodules: recursive

      - name: Install build dependencies
        run: |
          sudo apt-get update -q
          sudo apt-get install -y -q \
            clang-14 llvm-14 libelf-dev libbpf-dev \
            bpftool pahole \
            linux-headers-$(uname -r)
          sudo ln -sf /usr/bin/clang-14 /usr/bin/clang

      - name: Generate vmlinux.h
        run: bpftool btf dump file /sys/kernel/btf/vmlinux format c > src/vmlinux.h

      - name: Build BPF objects
        run: make bpf-objects

      - name: Build test binaries
        run: make tests

      - name: Upload build artifacts
        id: upload
        uses: actions/upload-artifact@v3
        with:
          name: bpf-test-binaries
          path: |
            tests/
            src/*.bpf.o

  run-tests-native:
    needs: build-bpf
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4

      - name: Download test binaries
        uses: actions/download-artifact@v3
        with:
          name: bpf-test-binaries

      - name: Install runtime dependencies
        run: |
          sudo apt-get install -y -q libbpf0 libelf1

      - name: Run BPF unit tests
        run: |
          sudo ./tests/test_my_prog \
            --json-output results/native-results.json

      - name: Upload test results
        uses: actions/upload-artifact@v3
        with:
          name: test-results-native
          path: results/

  run-tests-kernel-matrix:
    needs: build-bpf
    strategy:
      fail-fast: false
      matrix:
        kernel: ["5.15.0", "6.1.0", "6.6.0"]
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4

      - name: Download test binaries
        uses: actions/download-artifact@v3
        with:
          name: bpf-test-binaries

      - name: Set up kernel test environment
        run: |
          sudo apt-get install -y -q qemu-system-x86 \
            linux-image-${{ matrix.kernel }}-generic || \
          sudo scripts/build-test-kernel.sh ${{ matrix.kernel }}

      - name: Run tests in QEMU VM
        run: |
          sudo scripts/run-bpf-tests-in-vm.sh \
            /boot/vmlinuz-${{ matrix.kernel }} \
            ./tests/test_my_prog
        timeout-minutes: 15

      - name: Upload kernel matrix results
        uses: actions/upload-artifact@v3
        with:
          name: test-results-${{ matrix.kernel }}
          path: results/

Test Result Aggregation

Running tests across kernel versions produces multiple result sets. A simple aggregation script collects pass/fail counts and surfaces regressions:

#!/usr/bin/env python3
# scripts/aggregate-results.py
import json, sys, pathlib

results_dir = pathlib.Path("results")
totals = {"passed": 0, "failed": 0, "errors": []}

for result_file in results_dir.glob("**/*.json"):
    with open(result_file) as f:
        data = json.load(f)
    kernel = result_file.parent.name
    for test in data.get("tests", []):
        if test["status"] == "pass":
            totals["passed"] += 1
        else:
            totals["failed"] += 1
            totals["errors"].append({
                "kernel": kernel,
                "test": test["name"],
                "message": test.get("message", "")
            })

print(f"Passed: {totals['passed']}, Failed: {totals['failed']}")
if totals["errors"]:
    print("\nFailures:")
    for e in totals["errors"]:
        print(f"  [{e['kernel']}] {e['test']}: {e['message']}")
    sys.exit(1)

Verifier Regression Detection

One class of BPF CI failure is the verifier accepting a program on one kernel but rejecting it on another. This happens when the verifier gets more strict in a kernel update. Detect it by asserting on both success and failure cases:

/* Test that a known-safe program passes the verifier */
static void test_verifier_accepts_safe_prog(void)
{
    struct bpf_insn insns[] = {
        BPF_MOV64_IMM(BPF_REG_0, 0),
        BPF_EXIT_INSN(),
    };

    int prog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER,
                                "test_safe",
                                "GPL",
                                insns,
                                sizeof(insns) / sizeof(insns[0]),
                                NULL);
    if (prog_fd >= 0) {
        close(prog_fd);
        TEST_PASS("test_verifier_accepts_safe_prog");
    } else {
        TEST_FAIL("test_verifier_accepts_safe_prog",
                  "verifier rejected a trivially safe program");
    }
}

/* Test that a known-unsafe program is rejected */
static void test_verifier_rejects_unbounded_loop(void)
{
    struct bpf_insn insns[] = {
        BPF_MOV64_IMM(BPF_REG_0, 0),
        BPF_JMP_IMM(BPF_JA, 0, 0, -1),  /* infinite loop */
        BPF_EXIT_INSN(),
    };

    int prog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER,
                                "test_unsafe",
                                "GPL",
                                insns,
                                sizeof(insns) / sizeof(insns[0]),
                                NULL);
    if (prog_fd < 0) {
        TEST_PASS("test_verifier_rejects_unbounded_loop");
    } else {
        close(prog_fd);
        TEST_FAIL("test_verifier_rejects_unbounded_loop",
                  "verifier should have rejected this program");
    }
}

Integrating BPF CI with Application-Level Testing

BPF CI verifies the kernel layer of your stack. It tells you that your programs load, that maps behave correctly, and that no verifier regressions occurred. It does not tell you whether your observability data reaches your dashboard correctly, or whether your security policies are enforced end-to-end from a user's perspective.

That higher layer is where cloud-based test automation tools add value. Teams at companies running BPF-based security or observability products use HelpMeTest to cover the application tier — the control plane UI, the API that reads from BPF ring buffers, the alerting pipeline — while the BPF CI harness described here covers the kernel tier. HelpMeTest's AI-powered runner handles Robot Framework and Playwright tests at $100/month Pro, without requiring you to manage any additional infrastructure.

The two systems together give you kernel-to-UI coverage with each tier tested by tools appropriate to its environment.

Caching Kernel Builds

Building kernels in CI is slow — a full kernel build takes 20-40 minutes. Cache aggressively:

- name: Cache kernel build
  uses: actions/cache@v3
  with:
    path: kernel-builds/
    key: kernel-${{ matrix.kernel }}-${{ hashFiles('kernel-configs/bpf-test.config') }}
    restore-keys: |
      kernel-${{ matrix.kernel }}-

Pre-build kernels for your target versions and store them as release artifacts. Pull them at test time rather than building from source on every run.

Monitoring BPF CI Health

A BPF CI harness that flakes loses credibility fast. Monitor:

  • Flake rate — tests that sometimes pass and sometimes fail without code changes. Common causes: timing-dependent tracepoint delivery, kernel scheduling nondeterminism
  • QEMU timeout rate — VMs that hang instead of exiting. Add a watchdog timer in your init script
  • Artifact staleness — pre-built kernels and rootfs images drift from reality. Rebuild them on a weekly schedule

Track these metrics in your CI system's dashboard and treat flakes as bugs — fix them before they normalize.

Conclusion

A well-designed BPF CI harness catches verifier regressions, map logic errors, and kernel version incompatibilities before they reach production. The key components are: QEMU-based kernel VM isolation, a minimal rootfs with your test binaries, a kernel configuration matrix covering your supported versions, and structured test output for aggregation and trend tracking. Start with the native runner on the CI host kernel, add QEMU matrix testing once your native tests are stable, and treat every flake as a first-class bug. BPF programs run in the kernel's fast path — they deserve first-class testing infrastructure.

Read more