OpenXR

OpenXR Testing Strategies: Cross-Platform XR Test Automation

HelpMeTest

HelpMeTest

— 5 min read
OpenXR Testing Strategies: Cross-Platform XR Test Automation

OpenXR is the open standard for XR applications — the same code should work on Meta Quest, Valve Index, HTC Vive, and Windows Mixed Reality. In practice, runtime differences, driver bugs, and feature support gaps mean "write once, run anywhere" is an aspiration that requires systematic testing to achieve.

This guide covers testing strategies for OpenXR applications that catch cross-runtime compatibility issues before users do.

OpenXR Testing Landscape

OpenXR introduces testing concerns that don't exist in single-vendor SDKs:

  • Runtime conformance: Each vendor's OpenXR runtime has bugs and deviations from the spec
  • Action system portability: XR actions defined in your app must map correctly on each controller layout
  • Extension support: Optional extensions (hand tracking, passthrough, eye tracking) are supported differently across runtimes
  • Layer interactions: API layers (validation, compatibility shims) can affect behavior
  • Feature negotiation: Your app must gracefully handle missing optional features

Testing the OpenXR Action System

The OpenXR action system maps abstract input actions (like "grab") to specific controller inputs on each platform. Test that your action bindings are correct:

// test_action_system.cpp
#include <openxr/openxr.h>
#include <gtest/gtest.h>

class OpenXRActionTest : public ::testing::Test {
protected:
    XrInstance instance = XR_NULL_HANDLE;
    XrSession session = XR_NULL_HANDLE;
    XrActionSet actionSet = XR_NULL_HANDLE;
    
    void SetUp() override {
        // Create OpenXR instance (will use mock runtime in tests)
        XrInstanceCreateInfo createInfo = {XR_TYPE_INSTANCE_CREATE_INFO};
        strncpy(createInfo.applicationInfo.applicationName, "TestApp", XR_MAX_APPLICATION_NAME_SIZE);
        createInfo.applicationInfo.apiVersion = XR_CURRENT_API_VERSION;
        
        ASSERT_EQ(xrCreateInstance(&createInfo, &instance), XR_SUCCESS);
        
        // Create action set
        XrActionSetCreateInfo actionSetInfo = {XR_TYPE_ACTION_SET_CREATE_INFO};
        strncpy(actionSetInfo.actionSetName, "gameplay", XR_MAX_ACTION_SET_NAME_SIZE);
        strncpy(actionSetInfo.localizedActionSetName, "Gameplay", XR_MAX_LOCALIZED_ACTION_SET_NAME_SIZE);
        
        ASSERT_EQ(xrCreateActionSet(instance, &actionSetInfo, &actionSet), XR_SUCCESS);
    }
    
    void TearDown() override {
        if (actionSet != XR_NULL_HANDLE) xrDestroyActionSet(actionSet);
        if (instance != XR_NULL_HANDLE) xrDestroyInstance(instance);
    }
};

TEST_F(OpenXRActionTest, GrabAction_IsCreatedWithCorrectType) {
    XrAction grabAction = XR_NULL_HANDLE;
    XrActionCreateInfo actionInfo = {XR_TYPE_ACTION_CREATE_INFO};
    actionInfo.actionType = XR_ACTION_TYPE_FLOAT_INPUT;
    strncpy(actionInfo.actionName, "grab", XR_MAX_ACTION_NAME_SIZE);
    strncpy(actionInfo.localizedActionName, "Grab", XR_MAX_LOCALIZED_ACTION_NAME_SIZE);
    
    XrResult result = xrCreateAction(actionSet, &actionInfo, &grabAction);
    
    ASSERT_EQ(result, XR_SUCCESS);
    ASSERT_NE(grabAction, XR_NULL_HANDLE);
    
    xrDestroyAction(grabAction);
}

TEST_F(OpenXRActionTest, BoolAction_RejectsFloatSubpath) {
    XrAction boolAction = XR_NULL_HANDLE;
    XrActionCreateInfo actionInfo = {XR_TYPE_ACTION_CREATE_INFO};
    actionInfo.actionType = XR_ACTION_TYPE_BOOLEAN_INPUT;
    strncpy(actionInfo.actionName, "trigger_bool", XR_MAX_ACTION_NAME_SIZE);
    strncpy(actionInfo.localizedActionName, "Trigger Bool", XR_MAX_LOCALIZED_ACTION_NAME_SIZE);
    
    ASSERT_EQ(xrCreateAction(actionSet, &actionInfo, &boolAction), XR_SUCCESS);
    
    // Attempting to get float state from a bool action should fail
    XrActionStateFloat floatState = {XR_TYPE_ACTION_STATE_FLOAT};
    XrActionStateGetInfo getInfo = {XR_TYPE_ACTION_STATE_GET_INFO};
    getInfo.action = boolAction;
    
    // This should return XR_ERROR_ACTION_TYPE_MISMATCH
    XrResult result = xrGetActionStateFloat(session, &getInfo, &floatState);
    EXPECT_EQ(result, XR_ERROR_ACTION_TYPE_MISMATCH);
    
    xrDestroyAction(boolAction);
}

Testing OpenXR with Python (pyopenxr)

For higher-level behavior tests, Python's pyopenxr binding is easier to work with:

import pytest
import xr

class TestOpenXRSession:
    @pytest.fixture
    def xr_instance(self):
        instance = xr.create_instance(
            application_info=xr.ApplicationInfo(
                application_name="TestApp",
                application_version=xr.Version(1, 0, 0),
                api_version=xr.Version(1, 0, 0)
            ),
            enabled_extension_names=[]
        )
        yield instance
        xr.destroy_instance(instance)
    
    def test_enumerate_api_layers(self, xr_instance):
        """API layers can be enumerated without error."""
        layer_count, _ = xr.enumerate_api_layer_properties()
        assert layer_count >= 0  # May be zero, but should not error
    
    def test_instance_properties(self, xr_instance):
        """Instance properties return valid runtime info."""
        properties = xr.get_instance_properties(xr_instance)
        assert len(properties.runtime_name) > 0
        assert properties.runtime_version.major >= 1
    
    def test_system_properties_available(self, xr_instance):
        """System properties include expected XR capability flags."""
        try:
            system_id = xr.get_system(
                xr_instance,
                xr.SystemGetInfo(form_factor=xr.FormFactor.HEAD_MOUNTED_DISPLAY)
            )
            properties = xr.get_system_properties(xr_instance, system_id)
            
            # Basic graphics capabilities must be present
            assert properties.graphics_properties.max_swapchain_image_width > 0
            assert properties.graphics_properties.max_swapchain_image_height > 0
        except xr.exception.FormFactorUnavailableError:
            pytest.skip("No HMD form factor available in test environment")

Testing Extension Availability and Graceful Degradation

OpenXR extensions vary by runtime. Test that your app handles missing extensions correctly:

def test_hand_tracking_extension_handled_gracefully(xr_instance):
    """App works correctly when hand tracking extension is unavailable."""
    available_extensions = [
        ext.extension_name
        for ext in xr.enumerate_instance_extension_properties()
    ]
    
    if "XR_EXT_hand_tracking" not in available_extensions:
        # Extension unavailable — verify app doesn't crash and falls back correctly
        app = XRApplication(xr_instance, enable_hand_tracking=False)
        assert app.input_mode == InputMode.CONTROLLER, \
            "App should fall back to controller input when hand tracking unavailable"
    else:
        # Extension available — verify it initializes correctly
        app = XRApplication(xr_instance, enable_hand_tracking=True)
        assert app.hand_tracker is not None

def test_passthrough_extension_optional():
    """App launches successfully with or without passthrough extension."""
    # First, try without passthrough
    app_no_passthrough = XRApplicationBuilder() \
        .with_passthrough(enabled=False) \
        .build()
    assert app_no_passthrough.is_initialized
    
    # Verify UI elements visible without passthrough background
    assert app_no_passthrough.background_mode == BackgroundMode.SOLID

def test_missing_extension_does_not_crash_on_feature_use():
    """Calling a feature that requires an unavailable extension raises a clear error."""
    app = XRApplication()
    app.disable_extension("XR_EXT_eye_tracking")
    
    with pytest.raises(FeatureNotAvailableError, match="XR_EXT_eye_tracking"):
        app.start_eye_tracking()

Cross-Runtime Compatibility Testing

Test that your app behaves correctly across different OpenXR runtimes:

# pytest parameterization for runtime-specific tests
SUPPORTED_RUNTIMES = ["meta", "steamvr", "wmr", "monado"]

@pytest.mark.parametrize("runtime", SUPPORTED_RUNTIMES)
def test_controller_binding_valid_for_runtime(runtime):
    """Controller action binding works on each supported runtime."""
    binding_validator = ControllerBindingValidator()
    
    result = binding_validator.validate(
        action_manifest="bindings/default.json",
        runtime=runtime
    )
    
    assert result.is_valid, \
        f"Invalid controller bindings for {runtime}: {result.errors}"
    
    # All required actions must be bound
    required_actions = ["grab", "trigger", "thumbstick", "menu"]
    for action in required_actions:
        assert action in result.bound_actions, \
            f"Action '{action}' not bound for {runtime} runtime"

@pytest.mark.parametrize("runtime", SUPPORTED_RUNTIMES)
def test_session_state_transitions_on_runtime(runtime, mock_runtime):
    """Session state machine transitions correctly on each runtime."""
    session = XRSession(runtime=mock_runtime(runtime))
    
    # IDLE -> READY
    session.begin()
    assert session.state == XRSessionState.READY
    
    # READY -> SYNCHRONIZED  
    session.begin_frame()
    assert session.state in (XRSessionState.SYNCHRONIZED, XRSessionState.VISIBLE)
    
    # End gracefully
    session.end()
    assert session.state == XRSessionState.STOPPING

OpenXR Validation Layer Testing

The OpenXR validation layer catches spec violations. Use it in your CI:

import subprocess
import json

def test_no_openxr_validation_errors(xr_application_binary, tmp_path):
    """Application produces no OpenXR validation errors."""
    # Run with validation layer enabled
    result = subprocess.run(
        [xr_application_binary, "--test-mode", "--frames=100"],
        env={
            **os.environ,
            "XR_ENABLE_API_LAYERS": "XR_APILAYER_LUNARG_core_validation",
            "XR_API_LAYER_ENABLE_SETTINGS": "1"
        },
        capture_output=True,
        timeout=30
    )
    
    # Parse validation output
    stderr = result.stderr.decode("utf-8")
    
    errors = [
        line for line in stderr.splitlines()
        if "VUID-" in line or "Validation Error" in line
    ]
    
    assert not errors, \
        f"OpenXR validation errors detected:\n" + "\n".join(errors[:10])
    
    warnings = [
        line for line in stderr.splitlines()
        if "Validation Warning" in line
    ]
    
    # Warnings don't fail the test, but log them
    if warnings:
        print(f"OpenXR validation warnings:\n" + "\n".join(warnings))

CI/CD for OpenXR Tests

OpenXR tests that don't require hardware can run in CI using the Monado open-source runtime:

# .github/workflows/openxr-tests.yml
name: OpenXR Tests

on: [push, pull_request]

jobs:
  openxr-tests:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Install Monado (open-source OpenXR runtime)
        run: |
          sudo apt-get update
          sudo apt-get install -y libopenxr-dev libopenxr-loader1 monado-cli

      - name: Install Python dependencies
        run: pip install pytest pyopenxr

      - name: Run OpenXR logic tests (headless)
        run: |
          export XR_RUNTIME_JSON=/usr/share/openxr/1/openxr_monado.json
          pytest tests/openxr/ -v -k "not requires_headset"

      - name: Run OpenXR action system tests
        run: |
          export XR_RUNTIME_JSON=/usr/share/openxr/1/openxr_monado.json
          export DISPLAY=:99
          Xvfb :99 -screen 0 1024x768x24 &
          pytest tests/openxr/test_actions.py -v

Common OpenXR Testing Mistakes

Not testing extension unavailability: Most XR features are optional extensions. Test that your app degrades gracefully when an extension isn't available on a given runtime.

Runtime-specific tests only: If you only test on one runtime, you'll miss cross-platform compatibility bugs. Use a parametrized test matrix.

Ignoring the validation layer: The OpenXR validation layer catches spec violations that may work on one runtime but fail on another. Always run tests with validation enabled.

Hardcoded controller paths: /user/hand/right/input/trigger/value is correct for most controllers but not all. Test your action bindings on each target platform's controller layout.

No session lifecycle tests: OpenXR sessions have a complex state machine (IDLE → READY → SYNCHRONIZED → VISIBLE → FOCUSED → STOPPING → LOSS_PENDING). Test that your app handles all state transitions correctly.

OpenXR testing is largely about cross-runtime compatibility — ensuring that abstractions hold up across Meta, SteamVR, Windows MR, and mobile AR runtimes. The patterns here (action system validation, extension graceful degradation, runtime parametrization, validation layer testing) give you confidence that your OpenXR application will work across the ecosystem, not just on the device you developed it on.

Read more

Start now free