Testing

Testing Real-Time Operating Systems (RTOS): Tasks, Timing, and Concurrency

HelpMeTest

HelpMeTest

— 9 min read
Testing Real-Time Operating Systems (RTOS): Tasks, Timing, and Concurrency

Real-Time Operating Systems introduce concurrency, scheduling, and timing constraints that make testing significantly harder than single-threaded firmware. This guide covers testing strategies for FreeRTOS, Zephyr, and VxWorks: verifying task timing and deadlines, detecting race conditions and priority inversion, testing memory management in constrained environments, using SystemView for execution tracing, and running RTOS tests without physical hardware using QEMU and POSIX simulators.

Key Takeaways

RTOS bugs are often non-deterministic — timing-dependent failures don't reproduce on first retry. This makes systematic testing approaches (not ad-hoc debugging) essential. Tests must stress timing, not just logic.

Priority inversion is the most common and dangerous concurrency bug in RTOS. A low-priority task holding a mutex blocks a high-priority task. Test for it explicitly with carefully orchestrated task activation sequences.

Tick hooks and trace macros are non-invasive instrumentation. FreeRTOS and Zephyr both provide hook callbacks and trace point macros that let you measure timing without modifying task code.

Stack overflow detection requires both compile-time and runtime guards. FreeRTOS's configCHECK_FOR_STACK_OVERFLOW + vApplicationStackOverflowHook catches the most common cases; pattern canaries catch overflows that happen between scheduler ticks.

POSIX and QEMU ports of FreeRTOS and Zephyr enable host-based RTOS testing. You can run the full RTOS scheduler with real task preemption on a Linux host, eliminating the need for hardware for most unit and integration tests.

Why RTOS Testing Is Different

In a single-threaded bare-metal application, execution is deterministic: the CPU runs one instruction after another in a predictable order. You can reason about the state of the system at any point by reading the code.

An RTOS changes this completely. Multiple tasks run pseudo-concurrently, the scheduler preempts them at arbitrary points, interrupt service routines execute asynchronously, and shared resources must be protected by semaphores and mutexes. A bug may only appear when Task A preempts Task B at exactly the right moment — a moment that occurs once in a thousand runs.

This non-determinism makes RTOS bugs:

  • Hard to reproduce reliably in testing
  • Impossible to find by reading code alone
  • Potentially catastrophic in safety-critical applications
  • Sensitive to changes in unrelated code (because timing shifts)

Effective RTOS testing requires a structured approach that goes beyond "run it and see if it crashes."

RTOS Concepts Review

Task States and Scheduling

In FreeRTOS and most RTOSes, a task exists in one of these states: Running, Ready, Blocked (waiting on a semaphore/queue/delay), Suspended, or Deleted.

The scheduler runs the highest-priority Ready task. When tasks of equal priority exist, a round-robin time slice (one tick) applies. This means:

  • A high-priority task waiting on a semaphore is Blocked, and lower-priority tasks run
  • When the semaphore is released, the high-priority task becomes Ready and preempts immediately
/* FreeRTOS task creation */
TaskHandle_t sensor_task_handle;

xTaskCreate(
    sensor_task,        /* function */
    "SensorTask",       /* name */
    configMINIMAL_STACK_SIZE * 4,  /* stack depth in words */
    NULL,               /* parameters */
    tskIDLE_PRIORITY + 2,  /* priority */
    &sensor_task_handle
);

Synchronization Primitives

Primitive FreeRTOS API Use Case
Binary semaphore xSemaphoreCreateBinary Signal between ISR and task
Counting semaphore xSemaphoreCreateCounting Resource pool
Mutex xSemaphoreCreateMutex Mutual exclusion with priority inheritance
Recursive mutex xSemaphoreCreateRecursiveMutex Re-entrant locking
Queue xQueueCreate Typed data between tasks
Event group xEventGroupCreate Multiple flags, wait for any/all
Task notification xTaskNotify Lightweight signal, no allocation

The choice of primitive affects testability: queues are easier to test because you can send known values and assert on results; global variables protected by mutexes are harder because you must coordinate task scheduling to observe the protected section.

Testing Task Timing and Deadlines

Measuring Task Execution Time

FreeRTOS provides vTaskGetRunTimeStats() when configGENERATE_RUN_TIME_STATS is enabled. For precision timing, use a hardware timer with sub-microsecond resolution:

/* Configure a free-running 32-bit timer for profiling */
static volatile uint32_t profiling_timer_count = 0;

/* Called from a high-resolution timer ISR */
void profiling_timer_isr(void) {
    profiling_timer_count++;
}

uint32_t get_profiling_time_us(void) {
    return profiling_timer_count;  /* assumes 1µs tick */
}

/* Instrument a task */
void control_task(void *pvParameters) {
    uint32_t start, elapsed;
    TickType_t last_wake = xTaskGetTickCount();

    while (1) {
        start = get_profiling_time_us();

        /* Do work */
        run_control_loop();

        elapsed = get_profiling_time_us() - start;

        if (elapsed > CONTROL_LOOP_DEADLINE_US) {
            deadline_miss_count++;
            log_deadline_miss(elapsed);
        }

        vTaskDelayUntil(&last_wake, pdMS_TO_TICKS(10));  /* 10ms period */
    }
}

Testing Deadline Compliance

A test that verifies the control task meets its 10 ms deadline under load:

/* test_timing.c — Unity test */
#include "unity.h"
#include "FreeRTOS.h"
#include "task.h"
#include "profiling.h"

extern volatile uint32_t deadline_miss_count;

void test_control_loop_meets_deadline_under_load(void) {
    deadline_miss_count = 0;

    /* Start stress tasks at equal or lower priority */
    TaskHandle_t stress1, stress2;
    xTaskCreate(memory_stress_task, "Stress1", 512, NULL, tskIDLE_PRIORITY + 1, &stress1);
    xTaskCreate(can_tx_stress_task, "Stress2", 512, NULL, tskIDLE_PRIORITY + 1, &stress2);

    /* Run for 1 second */
    vTaskDelay(pdMS_TO_TICKS(1000));

    vTaskDelete(stress1);
    vTaskDelete(stress2);

    TEST_ASSERT_EQUAL_MESSAGE(0, deadline_miss_count,
        "Control loop missed deadline under load");
}

Zephyr Kernel Timing Tests

Zephyr provides k_cycle_get_32() and the timing subsystem for precision measurement:

#include <zephyr/kernel.h>
#include <zephyr/timing/timing.h>
#include <ztest.h>

ZTEST(timing_suite, test_isr_to_task_latency) {
    timing_t start, end;
    uint64_t latency_ns;

    timing_init();
    timing_start();

    /* Simulate: ISR fires, gives semaphore, measure time until task wakes */
    extern struct k_sem isr_sem;
    k_sem_reset(&isr_sem);

    start = timing_counter_get();
    k_sem_give(&isr_sem);  /* would normally be from ISR */

    /* task_under_test runs and records its start time */
    k_sleep(K_MSEC(5));

    end = timing_counter_get();
    latency_ns = timing_cycles_to_ns(timing_cycles_get(&start, &end));

    /* ISR-to-task latency should be under 50µs */
    zassert_true(latency_ns < 50000,
        "ISR-to-task latency %llu ns exceeds 50µs", latency_ns);
}

Race Condition and Priority Inversion Testing

Detecting Priority Inversion

Priority inversion occurs when a high-priority task waits on a mutex held by a low-priority task, which is itself preempted by a medium-priority task. The result: the high-priority task is effectively blocked by the medium-priority task — violating real-time guarantees.

FreeRTOS mutexes implement priority inheritance automatically: when a high-priority task blocks on a mutex, the holding task's priority is temporarily raised to match. Test that this mechanism works:

static SemaphoreHandle_t test_mutex;
static uint32_t low_task_run_count = 0;
static uint32_t high_task_blocked_ticks = 0;

void low_priority_task(void *pv) {
    /* Holds mutex for 100ms */
    xSemaphoreTake(test_mutex, portMAX_DELAY);
    TickType_t hold_start = xTaskGetTickCount();

    while ((xTaskGetTickCount() - hold_start) < pdMS_TO_TICKS(100)) {
        low_task_run_count++;
        taskYIELD();
    }

    xSemaphoreGive(test_mutex);
    vTaskDelete(NULL);
}

void medium_priority_task(void *pv) {
    /* Burns CPU at medium priority */
    while (1) {
        volatile uint32_t spin = 10000;
        while (spin--);
        taskYIELD();
    }
}

void high_priority_task(void *pv) {
    /* Wait for low-task to acquire mutex first */
    vTaskDelay(pdMS_TO_TICKS(10));

    TickType_t wait_start = xTaskGetTickCount();
    xSemaphoreTake(test_mutex, portMAX_DELAY);
    high_task_blocked_ticks = xTaskGetTickCount() - wait_start;
    xSemaphoreGive(test_mutex);
    vTaskDelete(NULL);
}

void test_priority_inheritance_limits_blocking(void) {
    test_mutex = xSemaphoreCreateMutex();

    /* Low: priority 1, Medium: priority 2, High: priority 3 */
    xTaskCreate(low_priority_task,    "Low",  512, NULL, 1, NULL);
    xTaskCreate(medium_priority_task, "Med",  512, NULL, 2, NULL);
    xTaskCreate(high_priority_task,   "High", 512, NULL, 3, NULL);

    vTaskDelay(pdMS_TO_TICKS(200));  /* let them run */

    /* With priority inheritance: High blocks for ~100ms (while Low holds mutex) */
    /* Without priority inheritance: High could block longer due to Med starving Low */
    TEST_ASSERT_LESS_THAN(pdMS_TO_TICKS(150), high_task_blocked_ticks);
}

Testing for Data Races

Data races are impossible to detect reliably through testing alone — but you can make them more likely to manifest by:

  1. Maximizing context switches around critical sections using taskYIELD() in both tasks
  2. Running with minimal tick rate (high context switch frequency)
  3. Using thread sanitizers when running POSIX ports on Linux (see below)
/* Stress test for a ring buffer shared between tasks */
void test_ring_buffer_concurrent_access(void) {
    ring_buffer_t rb;
    ring_buffer_init(&rb, 64);

    uint32_t write_errors = 0;
    uint32_t read_errors  = 0;

    /* Writer task */
    void writer(void *pv) {
        for (int i = 0; i < 10000; i++) {
            if (ring_buffer_write(&rb, (uint8_t)(i & 0xFF)) != RB_OK) {
                write_errors++;
            }
            taskYIELD();
        }
        vTaskDelete(NULL);
    }

    /* Reader task — verifies sequence is monotonically increasing */
    void reader(void *pv) {
        uint8_t prev = 0, curr;
        for (int i = 0; i < 10000; i++) {
            if (ring_buffer_read(&rb, &curr) == RB_OK) {
                if (curr != (uint8_t)(prev + 1) && prev != 0) {
                    read_errors++;
                }
                prev = curr;
            }
            taskYIELD();
        }
        vTaskDelete(NULL);
    }

    xTaskCreate(writer, "Writer", 512, NULL, tskIDLE_PRIORITY + 2, NULL);
    xTaskCreate(reader, "Reader", 512, NULL, tskIDLE_PRIORITY + 2, NULL);

    vTaskDelay(pdMS_TO_TICKS(5000));

    TEST_ASSERT_EQUAL_MESSAGE(0, read_errors, "Ring buffer sequence error (data race?)");
}

Stack Overflow Detection

Stack overflows are the most common memory safety bug in RTOS applications. FreeRTOS provides two detection methods.

Method 1: Paint and Check (configCHECK_FOR_STACK_OVERFLOW = 1)

FreeRTOS checks that the last 16 bytes of each stack still contain the known fill pattern (0xA5) at each context switch. Catches overflows that have stopped growing before the context switch.

Method 2: Pointer Check (configCHECK_FOR_STACK_OVERFLOW = 2)

Also checks that the stack pointer is within the valid range after each context switch. Catches active overflow in progress.

Enable in FreeRTOSConfig.h:

#define configCHECK_FOR_STACK_OVERFLOW  2
#define configUSE_MALLOC_FAILED_HOOK    1

/* Implement the hook */
void vApplicationStackOverflowHook(TaskHandle_t xTask, char *pcTaskName) {
    /* Log the overflowing task name before halting */
    fault_log("STACK OVERFLOW: task=%s", pcTaskName);
    __disable_irq();
    while (1);
}

Testing Stack Monitoring

Test that your monitoring infrastructure correctly detects and reports overflows:

void deliberate_overflow_task(void *pv) {
    /* Allocate more than the task's stack */
    volatile uint8_t large_array[2048];  /* task only has 512 words */
    memset((void *)large_array, 0xBB, sizeof(large_array));
    vTaskDelete(NULL);
}

void test_stack_overflow_detected(void) {
    extern volatile int stack_overflow_detected;
    stack_overflow_detected = 0;

    /* Create task with deliberately small stack */
    xTaskCreate(deliberate_overflow_task, "Overflow", 128, NULL, 1, NULL);
    vTaskDelay(pdMS_TO_TICKS(100));

    TEST_ASSERT_EQUAL_MESSAGE(1, stack_overflow_detected,
        "Stack overflow not detected by hook");
}

Memory Leak Detection in RTOS

FreeRTOS's heap allocators (heap_1 through heap_5) do not detect leaks. Track allocation manually:

/* Wrapper around pvPortMalloc for leak tracking */
#ifdef TRACK_ALLOCATIONS
static size_t total_allocated = 0;
static size_t total_freed = 0;
static size_t alloc_count = 0;

void *tracked_malloc(size_t size) {
    void *ptr = pvPortMalloc(size + sizeof(size_t));
    if (ptr) {
        *(size_t *)ptr = size;
        total_allocated += size;
        alloc_count++;
        return (uint8_t *)ptr + sizeof(size_t);
    }
    return NULL;
}

void tracked_free(void *ptr) {
    if (ptr) {
        uint8_t *base = (uint8_t *)ptr - sizeof(size_t);
        size_t size = *(size_t *)base;
        total_freed += size;
        vPortFree(base);
    }
}

size_t get_net_allocations(void) {
    return total_allocated - total_freed;
}
#endif

Test that a processing cycle returns heap to its initial state:

void test_protocol_handler_no_memory_leak(void) {
    size_t heap_before = xPortGetFreeHeapSize();

    for (int i = 0; i < 100; i++) {
        /* Simulate receiving and processing a complete message */
        uint8_t test_frame[] = {0x01, 0x02, 0x03, 0x04};
        protocol_process_frame(test_frame, sizeof(test_frame));
    }

    size_t heap_after = xPortGetFreeHeapSize();

    TEST_ASSERT_EQUAL_MESSAGE(heap_before, heap_after,
        "Protocol handler leaked heap memory");
}

SystemView: Execution Tracing

SEGGER SystemView is a real-time recording and visualization tool that captures every context switch, ISR entry/exit, and scheduler event with microsecond resolution. It requires a J-Link debug probe.

Integrating SystemView with FreeRTOS

/* In FreeRTOSConfig.h */
#define configUSE_TRACE_FACILITY        1
#define INCLUDE_xTaskGetCurrentTaskHandle 1

/* In your application init */
#include "SEGGER_SYSVIEW.h"
SEGGER_SYSVIEW_Conf();
SEGGER_SYSVIEW_Start();

Custom SystemView Events

Mark application events to correlate with task timing:

#include "SEGGER_SYSVIEW.h"

#define SYSVIEW_EVT_SENSOR_READ_START  (SEGGER_SYSVIEW_GET_MODULE_EVENTID(0, 0))
#define SYSVIEW_EVT_SENSOR_READ_END    (SEGGER_SYSVIEW_GET_MODULE_EVENTID(0, 1))

void sensor_read(void) {
    SEGGER_SYSVIEW_RecordVoid(SYSVIEW_EVT_SENSOR_READ_START);

    /* ... actual sensor read ... */

    SEGGER_SYSVIEW_RecordVoid(SYSVIEW_EVT_SENSOR_READ_END);
}

SystemView's timeline view shows exactly when each task ran, for how long, which ISRs fired, and where deadline misses occurred — making it invaluable for diagnosing timing problems that appear in testing.

Host-Based RTOS Testing Without Hardware

FreeRTOS POSIX Port

The FreeRTOS POSIX/Linux simulator runs the scheduler using POSIX threads and signals. Every task becomes a POSIX thread; context switching uses SIGUSR1. This means you can run the full RTOS scheduler — with real preemption — on a Linux development machine.

git clone https://github.com/FreeRTOS/FreeRTOS.git
<span class="hljs-built_in">cd FreeRTOS/FreeRTOS/Demo/Posix_GCC
make
./build/posix_demo

You can compile your RTOS application and test suite against the POSIX port and run it natively:

gcc -I FreeRTOS/Source/include \
    -I FreeRTOS/Source/portable/ThirdParty/GCC/Posix \
    -I FreeRTOS/Source/portable/ThirdParty/GCC/Posix/utils \
    FreeRTOS/Source/*.c \
    FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c \
    FreeRTOS/Source/portable/MemMang/heap_3.c \
    src/your_tasks.c \
    tests/test_your_tasks.c \
    -lpthread -o test_rtos
./test_rtos

With heap_3.c (wraps system malloc), the POSIX port is also compatible with AddressSanitizer and ThreadSanitizer:

gcc -fsanitize=thread -fsanitize=address ... -o test_rtos_tsan
./test_rtos_tsan

ThreadSanitizer will detect data races that the RTOS mutex fails to protect — which is exactly the class of bug most likely to cause field failures.

Zephyr Native POSIX Target

Zephyr's native_posix board compiles the entire Zephyr kernel, your application, and your test suite into a Linux executable:

west build -b native_posix tests/my_rtos_tests
./build/zephyr/zephyr.exe

Zephyr's Ztest framework integrates directly:

#include <ztest.h>

ZTEST_SUITE(scheduler_tests, NULL, NULL, NULL, NULL, NULL);

ZTEST(scheduler_tests, test_task_preempts_at_correct_priority) {
    /* ... test implementation ... */
    zassert_equal(expected_order, actual_order, "Task preemption order wrong");
}

Run with verbose output:

./build/zephyr/zephyr.exe -- -v

QEMU for Timer and ISR Testing

Some tests require real timer hardware behavior (rollover, compare match) that the POSIX port does not simulate. Use QEMU:

# Zephyr on QEMU Cortex-M3
west build -b qemu_cortex_m3 tests/timer_tests
west build -t run

FreeRTOS on QEMU ARM:

qemu-system-arm \
  -machine mps2-an385 \
  -kernel build/test_rtos.elf \
  -nographic \
  -semihosting \
  -semihosting-config enable=on,target=native

Concurrency Test Patterns

The barrier pattern. Use a counting semaphore initialized to zero to synchronize test tasks to a known starting point before exercising concurrent behavior:

SemaphoreHandle_t start_barrier;
start_barrier = xSemaphoreCreateCounting(2, 0);

void task_a(void *pv) {
    xSemaphoreGive(start_barrier);
    xSemaphoreTake(start_barrier, portMAX_DELAY);
    /* Now A and B start simultaneously */
    do_concurrent_work();
    vTaskDelete(NULL);
}

void task_b(void *pv) {
    xSemaphoreGive(start_barrier);
    xSemaphoreTake(start_barrier, portMAX_DELAY);
    do_concurrent_work();
    vTaskDelete(NULL);
}

The stress-then-verify pattern. Run N iterations of concurrent access under high CPU load, then verify invariants rather than inspecting every intermediate state.

The timeout-as-failure pattern. Any xSemaphoreTake with portMAX_DELAY in a test is a potential hang. Use bounded timeouts and assert that they did not expire:

BaseType_t result = xSemaphoreTake(result_sem, pdMS_TO_TICKS(1000));
TEST_ASSERT_EQUAL_MESSAGE(pdTRUE, result, "Test timed out — possible deadlock");

Conclusion

RTOS testing is hard precisely because the bugs it needs to catch — race conditions, priority inversions, deadline misses, stack overflows — only manifest under specific concurrent conditions. A solid RTOS test strategy combines deterministic unit tests on the POSIX simulator (with ThreadSanitizer for race detection), timing tests on QEMU or real hardware with hardware timer instrumentation, and SystemView-based tracing for postmortem analysis of failures that only appear on target. The investment in tooling pays off every time a concurrency bug is found in the test bench rather than in a shipped product.

Read more