Testing Firmware OTA Updates: Reliability, Rollback, and Safety

07 Oct 2026 — 6 min read

Over-the-air (OTA) firmware updates are one of the highest-risk operations in an IoT system's lifecycle. A bad update pushed to a fleet of 100,000 devices can brick them all simultaneously — devices that may be physically inaccessible in a factory, oil rig, or residential building. Unlike a web deployment that can be rolled back in seconds, a failed firmware update on an embedded device might require a field engineer with a JTAG programmer.

Testing OTA updates thoroughly isn't optional. This post covers what to test, how to test it, and how to catch failures before they reach production.

The OTA Update Process and Its Failure Points

A typical OTA update flows through these stages:

Distribution — server makes new firmware available
Discovery — device polls or receives notification of new version
Download — firmware binary transferred to device (may be chunked)
Verification — cryptographic signature and hash validation
Write — firmware written to alternate flash partition
Activation — device reboots into new partition
Health check — new firmware validates itself and confirms success
Commit — new firmware is marked permanent (old partition can be overwritten)

Every stage has failure modes. Your test suite needs to cover them.

Setting Up an OTA Test Environment

For embedded systems, a good OTA test environment includes:

A test MQTT broker or HTTP server serving the update manifest and binary
Hardware-in-the-loop (HIL) devices or QEMU emulation
A power controller to simulate brownouts during update
Serial/JTAG access to observe device state during the update

For devices using the ESP-IDF or Zephyr frameworks, you can run OTA tests in QEMU:

# Build firmware with OTA support for QEMU target
idf.py -B build_qemu set-target esp32 menuconfig  # enable OTA in sdkconfig
idf.py -B build_qemu build

# Run in QEMU with simulated OTA server
qemu-system-xtensa -nographic \
    -machine esp32 \
    -drive file=build_qemu/bootloader/bootloader.bin,if=mtd,format=raw \
    -drive file=build_qemu/partition_table/partition-table.bin,if=mtd,format=raw \
    -drive file=build_qemu/firmware.bin,if=mtd,format=raw \
    -serial tcp::5555,server,nowait \
    -net user,hostfwd=tcp::8443-:443

Testing Happy-Path OTA

The basic test: device on version N receives and applies version N+1 successfully.

import hashlib
import hmac
import json
import time
import pytest
from pathlib import Path

class OTATestServer:
    """Minimal HTTP server serving OTA manifests and binaries."""

    def __init__(self, firmware_dir: Path):
        self.firmware_dir = firmware_dir
        self.served_manifests = []
        self.download_log = []

    def get_manifest(self, current_version: str) -> dict:
        next_bin = self.firmware_dir / "firmware_v2.bin"
        binary = next_bin.read_bytes()
        sha256 = hashlib.sha256(binary).hexdigest()
        manifest = {
            "version": "2.0.0",
            "size": len(binary),
            "sha256": sha256,
            "url": f"http://ota-server/firmware/v2.0.0/firmware.bin",
            "rollback_supported": True,
            "min_space_kb": 512,
        }
        self.served_manifests.append({"requested_from": current_version, "served": manifest})
        return manifest


def test_successful_ota_update(device, ota_server):
    """Device on v1.0.0 should successfully update to v2.0.0."""
    assert device.get_firmware_version() == "1.0.0"

    # Trigger OTA check
    device.send_command("OTA_CHECK")
    time.sleep(2)

    # Device should discover the update
    status = device.get_ota_status()
    assert status["state"] == "downloading", f"Expected downloading, got: {status}"

    # Wait for download and verification (up to 120 seconds)
    deadline = time.time() + 120
    while time.time() < deadline:
        status = device.get_ota_status()
        if status["state"] in ("rebooting", "success", "error"):
            break
        time.sleep(2)

    assert status["state"] != "error", f"OTA failed: {status.get('error')}"

    # Wait for reboot and new version to be active
    time.sleep(15)
    device.wait_for_ready(timeout=30)

    new_version = device.get_firmware_version()
    assert new_version == "2.0.0", f"Expected v2.0.0, got {new_version}"

    # Verify device confirmed the update (prevent automatic rollback)
    ota_status = device.get_ota_status()
    assert ota_status["state"] == "committed", \
        "Update not committed — device may auto-rollback on next boot"

Testing Signature Verification

A firmware update that accepts unsigned or incorrectly signed binaries is a security catastrophe. Test that signature validation works:

import cryptography.hazmat.primitives.asymmetric.ed25519 as ed25519
from cryptography.hazmat.primitives.serialization import Encoding, Raw

def generate_test_keypair():
    private_key = ed25519.Ed25519PrivateKey.generate()
    public_key = private_key.public_key()
    return private_key, public_key

def sign_firmware(private_key, firmware_bytes: bytes) -> bytes:
    return private_key.sign(firmware_bytes)


def test_unsigned_firmware_rejected(device, ota_server):
    """Device must reject firmware with no signature."""
    ota_server.serve_firmware_without_signature("2.0.0")

    device.send_command("OTA_CHECK")
    time.sleep(60)  # Allow time for download attempt

    status = device.get_ota_status()
    assert status["state"] == "error"
    assert "signature" in status.get("error", "").lower(), \
        f"Expected signature error, got: {status}"

    # Device must still be on old version
    assert device.get_firmware_version() == "1.0.0"


def test_wrong_key_signature_rejected(device, ota_server, trusted_private_key):
    """Device must reject firmware signed with an untrusted key."""
    attacker_key, _ = generate_test_keypair()
    firmware_bytes = Path("firmware_v2.bin").read_bytes()
    bad_signature = sign_firmware(attacker_key, firmware_bytes)

    ota_server.serve_firmware_with_signature("2.0.0", firmware_bytes, bad_signature)

    device.send_command("OTA_CHECK")
    time.sleep(60)

    status = device.get_ota_status()
    assert status["state"] == "error"
    assert device.get_firmware_version() == "1.0.0"


def test_corrupted_firmware_rejected(device, ota_server, trusted_private_key):
    """Device must detect and reject bit-flipped firmware."""
    firmware_bytes = bytearray(Path("firmware_v2.bin").read_bytes())
    firmware_bytes[1000] ^= 0xFF  # flip some bits mid-binary
    corrupted = bytes(firmware_bytes)

    # Sign the original (correct) hash — not the corrupted bytes
    original = Path("firmware_v2.bin").read_bytes()
    good_signature = sign_firmware(trusted_private_key, original)

    ota_server.serve_firmware_with_signature("2.0.0", corrupted, good_signature)

    device.send_command("OTA_CHECK")
    time.sleep(60)

    status = device.get_ota_status()
    assert status["state"] == "error"
    assert "hash" in status.get("error", "").lower() or "integrity" in status.get("error", "").lower()

Testing Rollback on Boot Failure

The most critical OTA feature is automatic rollback. If new firmware boots but fails its health check, the device must revert to the previous version automatically. Many teams assume this works without testing it.

def test_rollback_when_new_firmware_fails_health_check(device, ota_server):
    """If new firmware fails self-test, device must auto-rollback to previous version."""
    # Serve firmware that boots but deliberately fails its health check
    ota_server.serve_firmware_that_fails_health_check("2.0.0-bad")

    assert device.get_firmware_version() == "1.0.0"
    device.send_command("OTA_CHECK")

    # Wait for update download and reboot
    device.wait_for_reboot(timeout=120)
    device.wait_for_ready(timeout=30)

    time.sleep(10)  # Let health check run and fail

    # Device should have detected failure and rebooted back to 1.0.0
    device.wait_for_reboot(timeout=60)
    device.wait_for_ready(timeout=30)

    final_version = device.get_firmware_version()
    assert final_version == "1.0.0", \
        f"Expected rollback to 1.0.0, but device is running {final_version}"

    # Verify the failed update is not retried automatically
    time.sleep(30)
    assert device.get_firmware_version() == "1.0.0", \
        "Device attempted to re-apply the bad update after rollback"


def test_rollback_after_brownout_during_write(device, ota_server, power_controller):
    """Device must recover cleanly if power is cut during firmware write."""
    ota_server.serve_firmware("2.0.0")

    device.send_command("OTA_CHECK")

    # Wait until download starts, then cut power during the write phase
    deadline = time.time() + 30
    while time.time() < deadline:
        status = device.get_ota_status()
        if status["state"] == "writing" and status.get("progress_pct", 0) > 30:
            # Cut power mid-write
            power_controller.cut_power(duration_ms=500)
            break
        time.sleep(0.5)

    # Device should boot back to the original firmware
    device.wait_for_ready(timeout=60)
    version = device.get_firmware_version()
    assert version == "1.0.0", \
        f"Device did not recover to 1.0.0 after brownout during write, running {version}"

Testing Interrupted Downloads

Devices lose connectivity mid-download. Your OTA implementation should resume rather than restart:

def test_download_resumes_after_interruption(device, ota_server, network_controller):
    """OTA download should resume from checkpoint, not restart from zero."""
    ota_server.serve_large_firmware("2.0.0", size_kb=512)

    device.send_command("OTA_CHECK")

    # Wait until 50% downloaded, then interrupt
    deadline = time.time() + 60
    while time.time() < deadline:
        status = device.get_ota_status()
        if status.get("progress_pct", 0) >= 50:
            network_controller.disconnect()
            break
        time.sleep(0.5)

    time.sleep(5)  # Let device notice the disconnect

    download_bytes_before = device.get_ota_status().get("bytes_downloaded", 0)

    # Restore network
    network_controller.connect()

    # Monitor download restart
    time.sleep(10)
    download_bytes_after = device.get_ota_status().get("bytes_downloaded", 0)

    # Should resume, not restart (bytes_after should be >= bytes_before)
    assert download_bytes_after >= download_bytes_before, \
        "Download restarted from zero instead of resuming from checkpoint"

    # Wait for completion
    deadline = time.time() + 120
    while time.time() < deadline:
        status = device.get_ota_status()
        if status["state"] in ("committed", "error"):
            break
        time.sleep(2)

    assert device.get_firmware_version() == "2.0.0"

Testing Fleet-Scale OTA

Pushing to 10,000 devices simultaneously is a different problem than pushing to one. Test your rollout strategy:

def test_staged_rollout_stops_on_error_rate(fleet_manager, ota_server):
    """A staged rollout should halt if error rate exceeds threshold."""
    # Configure a 5% canary rollout
    rollout = fleet_manager.create_rollout(
        firmware_version="2.0.0",
        strategy="staged",
        stages=[
            {"percent": 5,  "wait_minutes": 10, "error_threshold_pct": 2},
            {"percent": 25, "wait_minutes": 30, "error_threshold_pct": 2},
            {"percent": 100},
        ]
    )

    # Simulate: 3 out of 5 canary devices fail (60% error rate)
    ota_server.configure_failure_rate("2.0.0", 0.6)

    rollout.start()
    time.sleep(15 * 60)  # wait past the stage 1 window

    status = rollout.get_status()
    assert status["state"] == "halted", \
        f"Rollout should have halted but state is: {status['state']}"
    assert status["halted_at_stage"] == 1
    assert status["devices_on_new_firmware_pct"] < 10, \
        f"Too many devices updated before halt: {status['devices_on_new_firmware_pct']}%"

Integrating OTA Tests Into CI/CD

# .github/workflows/ota-tests.yml
name: OTA Update Tests

on:
  push:
    paths:
      - 'firmware/**'
      - 'ota_server/**'
      - 'tests/ota/**'

jobs:
  ota-emulation-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build firmware images
        run: |
          make firmware VERSION=1.0.0 OUT=test/fixtures/firmware_v1.bin
          make firmware VERSION=2.0.0 OUT=test/fixtures/firmware_v2.bin

      - name: Start OTA test infrastructure
        run: docker compose -f docker-compose.ota-test.yml up -d

      - name: Run OTA happy-path tests
        run: pytest tests/ota/test_happy_path.py -v --timeout=300

      - name: Run OTA security tests
        run: pytest tests/ota/test_security.py -v --timeout=120

      - name: Run OTA resilience tests
        run: pytest tests/ota/test_resilience.py -v --timeout=600

  ota-hardware-tests:
    runs-on: [self-hosted, hil-rig]
    needs: ota-emulation-tests
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      - name: Flash baseline firmware
        run: scripts/flash_baseline.sh
      - name: Run HIL OTA tests
        run: pytest tests/ota/test_hil.py -v --timeout=900

Key Takeaways

Test every stage of the OTA pipeline, not just the happy path
Signature validation tests are mandatory — unsigned firmware acceptance is a critical security flaw
Test automatic rollback explicitly — never assume it works because the framework supports it
Simulate power loss during firmware write — this is the most common real-world failure mode
Test download resumption — devices on cellular connections drop and reconnect constantly
Test staged rollouts — a fleet-wide push without canary validation is an incident waiting to happen

The cost of thorough OTA testing is a few days of engineering work. The cost of skipping it is a bricked fleet.