OTA Firmware Update Testing
Over-the-air firmware updates are one of the highest-risk operations an embedded device performs. A failed update that leaves a device in an unbootable state — especially at scale — is an operational catastrophe. Yet most teams test OTA by manually pushing one update to one device and watching it reboot successfully. That is not a test. It is a hope.
This guide covers systematic OTA testing: validating update download integrity, signature verification, the rollback mechanism, power-loss recovery, and running these scenarios automatically in CI and on real hardware.
The OTA Update State Machine
Before writing tests, model the states your OTA system must handle:
IDLE
│
▼ Update available
DOWNLOADING
│ ✓ Download complete ✗ Checksum fail / network error
▼ │
VERIFYING ◄────────────────────── DOWNLOAD_FAILED (retry / abort)
│ ✓ Signature valid ✗ Signature invalid
▼ │
STAGING ◄──────────────────────── VERIFICATION_FAILED (abort, alert)
│ ✓ Written to inactive slot
▼
PENDING_REBOOT
│ Reboot triggered
▼
BOOTING_NEW
│ ✓ New firmware boots, marks itself valid
▼
COMMIT ────────────────────────► RUNNING_NEW
│ ✗ Boot fails or watchdog expires
▼
ROLLBACK ──────────────────────► RUNNING_OLD (revert partition, alert)Every transition is a test scenario. Every error path is a test scenario. Build your test suite from this state machine.
Testing Download Integrity
The first layer of defense is a checksum or hash over the downloaded binary. Test three cases: valid download, corrupted download, and truncated download.
A useful pattern is a local HTTP server that serves firmware images — either legitimate or deliberately corrupted — so tests run without a real update server:
import hashlib
import http.server
import threading
import struct
import pytest
def sha256_of_file(path: str) -> str:
h = hashlib.sha256()
with open(path, 'rb') as f:
for chunk in iter(lambda: f.read(65536), b''):
h.update(chunk)
return h.hexdigest()
class FirmwareServer:
"""Minimal HTTP server that serves firmware images for testing."""
def __init__(self, host='127.0.0.1', port=8080):
self.host = host
self.port = port
self._thread = None
self._server = None
def serve_file(self, path: str, corrupt: bool = False, truncate_at: int = None):
with open(path, 'rb') as f:
data = bytearray(f.read())
if corrupt:
# Flip a byte in the middle of the image
mid = len(data) // 2
data[mid] ^= 0xFF
if truncate_at is not None:
data = data[:truncate_at]
self._data = bytes(data)
self._sha256 = hashlib.sha256(self._data).hexdigest()
def start(self):
server_data = self._data
server_hash = self._sha256
class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self):
if self.path == '/firmware.bin':
self.send_response(200)
self.send_header('Content-Length', str(len(server_data)))
self.send_header('X-SHA256', server_hash)
self.end_headers()
self.wfile.write(server_data)
elif self.path == '/manifest.json':
import json
manifest = {'version': '2.1.0',
'url': f'http://127.0.0.1:8080/firmware.bin',
'sha256': server_hash,
'size': len(server_data)}
body = json.dumps(manifest).encode()
self.send_response(200)
self.send_header('Content-Length', str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *args):
pass # Suppress access logs during tests
self._server = http.server.HTTPServer((self.host, self.port), Handler)
self._thread = threading.Thread(target=self._server.serve_forever, daemon=True)
self._thread.start()
def stop(self):
if self._server:
self._server.shutdown()Tests then point the device at this local server:
@pytest.fixture
def fw_server(valid_firmware_path):
server = FirmwareServer()
server.serve_file(valid_firmware_path)
server.start()
yield server
server.stop()
def test_valid_download_accepted(fw_server, device_harness):
device_harness.set_update_server('http://127.0.0.1:8080')
result = device_harness.trigger_ota_check()
assert result['download_status'] == 'SUCCESS'
assert result['sha256_match'] == True
def test_corrupted_download_rejected(valid_firmware_path, device_harness):
server = FirmwareServer()
server.serve_file(valid_firmware_path, corrupt=True)
server.start()
device_harness.set_update_server('http://127.0.0.1:8080')
result = device_harness.trigger_ota_check()
assert result['download_status'] == 'CHECKSUM_FAIL'
assert result['ota_state'] == 'DOWNLOAD_FAILED'
assert result['current_version'] == device_harness.original_version, \
"Device must not advance to verification after checksum failure"
server.stop()
def test_truncated_download_triggers_retry(valid_firmware_path, device_harness):
image_size = os.path.getsize(valid_firmware_path)
server = FirmwareServer()
server.serve_file(valid_firmware_path, truncate_at=image_size // 2)
server.start()
device_harness.set_update_server('http://127.0.0.1:8080')
result = device_harness.trigger_ota_check()
assert result['download_status'] in ('CHECKSUM_FAIL', 'SIZE_MISMATCH')
server.stop()Testing Signature Verification
Download integrity (SHA-256) confirms the file arrived uncorrupted. Signature verification (ECDSA or RSA) confirms the file came from your organization. These are separate checks and both need tests:
def test_unsigned_firmware_rejected(unsigned_firmware_path, device_harness):
"""Device must refuse to stage firmware without a valid signature."""
server = FirmwareServer()
server.serve_file(unsigned_firmware_path)
server.start()
device_harness.set_update_server('http://127.0.0.1:8080')
result = device_harness.trigger_ota_check()
assert result['verification_status'] == 'SIGNATURE_INVALID'
assert result['ota_state'] == 'VERIFICATION_FAILED'
assert result['current_version'] == device_harness.original_version
server.stop()
def test_firmware_signed_with_wrong_key_rejected(wrong_key_firmware_path, device_harness):
"""Device must reject firmware signed with any key other than the provisioned public key."""
server = FirmwareServer()
server.serve_file(wrong_key_firmware_path)
server.start()
device_harness.set_update_server('http://127.0.0.1:8080')
result = device_harness.trigger_ota_check()
assert result['verification_status'] == 'SIGNATURE_INVALID'
server.stop()
def test_rollback_to_older_version_rejected(older_signed_firmware, device_harness):
"""Anti-rollback: device must not accept valid firmware with version < current."""
server = FirmwareServer()
server.serve_file(older_signed_firmware)
server.start()
device_harness.set_update_server('http://127.0.0.1:8080')
result = device_harness.trigger_ota_check()
assert result['verification_status'] == 'VERSION_DOWNGRADE_REJECTED'
server.stop()Testing Rollback on Boot Failure
The rollback mechanism is the most critical part of any OTA system. An update that bricks devices in the field is worse than no update at all. Test that the device boots back to the previous firmware when the new image fails its self-test:
def test_rollback_on_watchdog_timeout(device_harness, fw_server_with_bad_firmware):
"""If new firmware triggers watchdog on first boot, device must roll back."""
original_version = device_harness.get_running_version()
device_harness.set_update_server(fw_server_with_bad_firmware.url)
device_harness.trigger_ota_check()
# Wait for the device to download, verify, stage, and reboot
device_harness.wait_for_reboot(timeout=60)
# New firmware crashes and triggers watchdog
# Device should reboot again into old firmware
device_harness.wait_for_reboot(timeout=30) # Second reboot: rollback
recovered_version = device_harness.get_running_version()
assert recovered_version == original_version, \
f"Device should have rolled back to {original_version}, running {recovered_version}"
ota_state = device_harness.get_ota_state()
assert ota_state == 'ROLLBACK_COMPLETE'The "bad firmware" image in this test is a special test build that intentionally hangs in an infinite loop without feeding the watchdog, triggering a reset. The boot counter (typically stored in RTC backup registers or a dedicated sector of non-volatile memory) increments on each failed boot attempt and triggers rollback after a configurable threshold.
Testing Power-Loss Recovery
Power loss during OTA is one of the hardest failure modes to test manually but one of the most important to get right. A relay or FET controlled by the test host can cut power at precise moments:
def test_power_loss_during_download_leaves_old_firmware_intact(
device_harness, fw_server, power_relay):
"""Power loss mid-download must not corrupt the running firmware partition."""
original_version = device_harness.get_running_version()
device_harness.set_update_server(fw_server.url)
device_harness.start_ota_async()
# Wait until download is 50% complete, then cut power
device_harness.wait_for_download_progress(50, timeout=30)
power_relay.cut_power()
time.sleep(0.5)
power_relay.restore_power()
device_harness.wait_for_boot(timeout=30)
# Device must be running original firmware
running = device_harness.get_running_version()
assert running == original_version, \
"Power loss during download corrupted primary partition"
# OTA state must indicate incomplete download, not staged update
ota_state = device_harness.get_ota_state()
assert ota_state in ('IDLE', 'DOWNLOAD_FAILED'), \
f"Unexpected OTA state after power-loss recovery: {ota_state}"
def test_power_loss_during_flash_write_triggers_rollback(
device_harness, fw_server, power_relay):
"""Power loss while writing new partition must trigger rollback on next boot."""
original_version = device_harness.get_running_version()
device_harness.set_update_server(fw_server.url)
device_harness.start_ota_async()
device_harness.wait_for_ota_state('STAGING', timeout=60)
# Cut power during the flash write
power_relay.cut_power()
time.sleep(0.2)
power_relay.restore_power()
device_harness.wait_for_boot(timeout=30)
running = device_harness.get_running_version()
assert running == original_version, \
"Device should be on original firmware after power-loss during staging"The power relay is a simple GPIO-controlled MOSFET. The test host drives the GPIO to cut and restore the target's power supply. This test infrastructure is simple to build and catches entire classes of bugs that are impossible to find any other way.
Automating OTA Tests in CI
OTA tests are slower than unit tests — a full download, verify, flash, and reboot cycle can take several minutes. Organize them in a separate CI stage that runs on each build but does not block fast unit test feedback:
name: OTA Update Tests
on:
push:
branches: [main]
jobs:
ota-tests:
runs-on: [self-hosted, ota-bench]
timeout-minutes: 30
steps:
- uses: actions/checkout@v3
- name: Build firmware images
run: make firmware-release.bin firmware-test-bad.bin firmware-old.bin
- name: Sign firmware
run: ./scripts/sign_firmware.sh firmware-release.bin
- name: Flash baseline firmware
run: openocd -f target.cfg -c "program firmware-old.bin verify reset exit"
- name: Run OTA tests
run: pytest tests/ota/ -v --timeout=120 --html=ota-report.html
- name: Upload report
uses: actions/upload-artifact@v3
with:
name: ota-test-report
path: ota-report.htmlContinuous Monitoring After Deployment
Test automation verifies OTA behavior before release. Once an update rolls out to a fleet, you need real-time visibility into the rollout. HelpMeTest can monitor the device management API endpoints that track update status — version distribution across your fleet, rollback rates, devices stuck in update state — and alert you the moment a metric deviates from expected patterns. Catching a 5% rollback rate on a new firmware release within the first hour is the difference between a targeted rollback and a widespread outage.
OTA firmware updates are too consequential to test manually. Build the bench, write the failure scenarios, and run them on every release. The cost of the test infrastructure is trivial compared to the cost of a bricked fleet.
