Testing

Simulation-Based ADAS Testing with CARLA and LGSVL

HelpMeTest

HelpMeTest

— 8 min read
Simulation-Based ADAS Testing with CARLA and LGSVL

Road testing autonomous driving systems is expensive, slow, and dangerous for edge cases. Waymo famously ran over 20 billion simulated miles before reaching 20 million real miles. Simulation isn't a shortcut — it's the only way to test rare scenarios like sensor occlusion in a snowstorm, or a child darting between parked cars, at the scale needed to validate safety.

This post is a practical guide to ADAS simulation testing using CARLA and LGSVL: setting them up, writing Python scenario scripts, testing sensor fusion, and integrating simulation runs into CI.

Choosing Between CARLA and LGSVL

CARLA (Car Learning to Act): Open-source, Unreal Engine 4, strong Python API, large scenario library, active research community. Best for algorithmic testing and sensor model experimentation.

LGSVL (now SVL Simulator): Unity-based, better ROS 2 and Autoware integration, higher visual fidelity for perception testing, supports Apollo. Better for full-stack autonomy stack integration.

Use CARLA for unit-level perception and planning tests. Use LGSVL for end-to-end stack validation against Autoware/Apollo.

CARLA Setup

# Install CARLA 0.9.15
wget https://carla-releases.s3.eu-west-3.amazonaws.com/Linux/CARLA_0.9.15.tar.gz
tar -xf CARLA_0.9.15.tar.gz -C /opt/carla

# Install Python client
pip install carla==0.9.15

# Start the server (headless for CI)
/opt/carla/CarlaUE4.sh -RenderOffScreen -quality-level=Low -world-port=2000 &
sleep 10  # Wait for server initialization

# Verify connection
python3 -c "
import carla
client = carla.Client('localhost', 2000)
client.set_timeout(10.0)
world = client.get_world()
print(f'Connected. Map: {world.get_map().name}')
"

Writing CARLA Scenarios for ADAS Testing

The CARLA Python API gives you full control over the simulation: spawning actors, controlling weather, injecting sensor noise, triggering pedestrian behaviors.

Scenario: Pedestrian Crossing — AEB Validation

#!/usr/bin/env python3
"""
AEB (Automatic Emergency Braking) validation scenario.
Tests: Does the ADAS system detect and respond to a pedestrian
crossing from behind a parked vehicle?

Pass criteria:
  - Vehicle decelerates to < 5 kph before reaching pedestrian position
  - No false trigger on scenario variants without pedestrian
  - Response latency < 300ms from pedestrian visibility
"""

import carla
import time
import math
import numpy as np
from dataclasses import dataclass
from typing import Optional

@dataclass
class ScenarioResult:
    passed: bool
    min_speed_at_crossing_kph: float
    aeb_trigger_latency_ms: Optional[float]
    collision_occurred: bool

class PedestrianCrossingScenario:
    def __init__(self, host: str = 'localhost', port: int = 2000):
        self.client = carla.Client(host, port)
        self.client.set_timeout(10.0)
        self.world = self.client.get_world()
        self.blueprint_library = self.world.get_blueprint_library()
        self.actors = []

    def _cleanup(self):
        for actor in self.actors:
            if actor.is_alive:
                actor.destroy()
        self.actors.clear()

    def run(self, ego_speed_kph: float = 50.0,
            pedestrian_hidden: bool = True) -> ScenarioResult:
        """
        Args:
            ego_speed_kph: Initial ego vehicle speed
            pedestrian_hidden: If True, pedestrian starts hidden behind parked car
        """
        try:
            return self._run_scenario(ego_speed_kph, pedestrian_hidden)
        finally:
            self._cleanup()

    def _run_scenario(self, ego_speed_kph: float,
                      pedestrian_hidden: bool) -> ScenarioResult:
        map_obj = self.world.get_map()
        spawn_points = map_obj.get_spawn_points()

        # Spawn ego vehicle on Town03 straight road
        ego_bp = self.blueprint_library.find('vehicle.tesla.model3')
        ego_transform = carla.Transform(
            carla.Location(x=-50, y=0, z=0.5),
            carla.Rotation(yaw=0)
        )
        ego = self.world.spawn_actor(ego_bp, ego_transform)
        self.actors.append(ego)

        # Spawn parked vehicle (occlusion object)
        parked_bp = self.blueprint_library.find('vehicle.audi.a2')
        parked_transform = carla.Transform(
            carla.Location(x=20, y=3.5, z=0.5),  # On the shoulder
            carla.Rotation(yaw=0)
        )
        parked = self.world.spawn_actor(parked_bp, parked_transform)
        parked.apply_control(carla.VehicleControl(hand_brake=True))
        self.actors.append(parked)

        # Spawn pedestrian
        ped_bp = self.blueprint_library.find('walker.pedestrian.0001')
        ped_start_y = 3.5 if pedestrian_hidden else 0.0
        ped_transform = carla.Transform(
            carla.Location(x=25, y=ped_start_y, z=0.5)
        )
        pedestrian = self.world.spawn_actor(ped_bp, ped_transform)
        self.actors.append(pedestrian)

        # Set ego vehicle to target speed
        ego.enable_constant_velocity(
            carla.Vector3D(ego_speed_kph / 3.6, 0, 0)
        )

        # Attach collision sensor to ego
        collision_bp = self.blueprint_library.find('sensor.other.collision')
        collision_sensor = self.world.spawn_actor(
            collision_bp, carla.Transform(), attach_to=ego
        )
        self.actors.append(collision_sensor)

        collision_occurred = False
        def on_collision(event):
            nonlocal collision_occurred
            collision_occurred = True
        collision_sensor.listen(on_collision)

        # Simulation loop: 10 seconds max
        pedestrian_visible_at = None
        aeb_triggered_at = None
        min_speed = ego_speed_kph
        speeds = []

        controller_bp = self.blueprint_library.find('controller.ai.walker')
        controller = self.world.spawn_actor(controller_bp, carla.Transform(),
                                             attach_to=pedestrian)
        self.actors.append(controller)
        controller.start()

        start = time.time()
        pedestrian_crossing_triggered = False

        while time.time() - start < 10.0:
            self.world.tick()

            ego_loc = ego.get_location()
            ped_loc = pedestrian.get_location()

            # Trigger pedestrian to cross when ego is 15m away
            if not pedestrian_crossing_triggered and ego_loc.x > 10:
                controller.go_to_location(
                    carla.Location(x=25, y=-3.5, z=0.5)
                )
                controller.set_max_speed(1.4)  # 1.4 m/s walking speed
                pedestrian_crossing_triggered = True

            # Check if pedestrian is in ego's sensor FOV (simplified: within 30m, -15° to 15° azimuth)
            dx = ped_loc.x - ego_loc.x
            dy = ped_loc.y - ego_loc.y
            distance = math.sqrt(dx**2 + dy**2)
            azimuth_deg = math.degrees(math.atan2(dy, dx))

            if distance < 30 and abs(azimuth_deg) < 15 and pedestrian_visible_at is None:
                pedestrian_visible_at = time.time()

            # Read ego speed
            v = ego.get_velocity()
            speed_kph = math.sqrt(v.x**2 + v.y**2) * 3.6
            speeds.append(speed_kph)
            min_speed = min(min_speed, speed_kph)

            # Detect AEB trigger (speed drop > 20% within 500ms)
            if (pedestrian_visible_at and aeb_triggered_at is None
                    and len(speeds) > 5 and speed_kph < ego_speed_kph * 0.8):
                aeb_triggered_at = time.time()

            # Stop simulation if ego passed crossing point
            if ego_loc.x > 40:
                break

        latency_ms = None
        if pedestrian_visible_at and aeb_triggered_at:
            latency_ms = (aeb_triggered_at - pedestrian_visible_at) * 1000

        passed = (
            min_speed < 5.0  # Nearly stopped before pedestrian
            and not collision_occurred
            and (latency_ms is None or latency_ms < 300)
        )

        return ScenarioResult(
            passed=passed,
            min_speed_at_crossing_kph=min_speed,
            aeb_trigger_latency_ms=latency_ms,
            collision_occurred=collision_occurred
        )


if __name__ == '__main__':
    scenario = PedestrianCrossingScenario()

    # Test 1: Standard crossing with occlusion
    result = scenario.run(ego_speed_kph=50.0, pedestrian_hidden=True)
    print(f"Occluded crossing: {'PASS' if result.passed else 'FAIL'}")
    print(f"  Min speed: {result.min_speed_at_crossing_kph:.1f} kph")
    print(f"  AEB latency: {result.aeb_trigger_latency_ms:.0f} ms")
    print(f"  Collision: {result.collision_occurred}")

    # Test 2: No pedestrian — verify no false trigger
    result2 = scenario.run(ego_speed_kph=50.0, pedestrian_hidden=False)
    assert result2.aeb_trigger_latency_ms is None, "False AEB trigger!"
    print("No-pedestrian variant: PASS (no false trigger)")

Sensor Fusion Testing: Camera + LiDAR + Radar

A key ADAS testing concern is sensor fusion correctness — ensuring the fusion algorithm correctly combines inputs from heterogeneous sensors. CARLA provides synthetic ground truth you can use to measure fusion accuracy.

import carla
import numpy as np
import queue

class SensorFusionTestBench:
    """
    Validates fusion algorithm by comparing fused output to CARLA ground truth.
    Metrics:
      - Detection rate: What % of GT objects appear in fused output?
      - False positive rate: What % of fused detections have no GT match?
      - Position error: Mean distance between matched GT and fused objects
    """

    def setup_sensors(self, ego: carla.Actor):
        bp_lib = self.world.get_blueprint_library()

        # Camera (forward-facing, 90° FOV)
        cam_bp = bp_lib.find('sensor.camera.rgb')
        cam_bp.set_attribute('image_size_x', '1920')
        cam_bp.set_attribute('image_size_y', '1080')
        cam_bp.set_attribute('fov', '90')
        camera = self.world.spawn_actor(
            cam_bp,
            carla.Transform(carla.Location(x=2.0, z=1.4)),
            attach_to=ego
        )

        # LiDAR (Velodyne HDL-64E equivalent)
        lidar_bp = bp_lib.find('sensor.lidar.ray_cast')
        lidar_bp.set_attribute('channels', '64')
        lidar_bp.set_attribute('range', '100.0')
        lidar_bp.set_attribute('points_per_second', '1300000')
        lidar_bp.set_attribute('rotation_frequency', '10')
        lidar_bp.set_attribute('upper_fov', '2')
        lidar_bp.set_attribute('lower_fov', '-24.8')
        lidar = self.world.spawn_actor(
            lidar_bp,
            carla.Transform(carla.Location(x=0.0, z=1.8)),
            attach_to=ego
        )

        # Radar (forward-facing, 35° azimuth)
        radar_bp = bp_lib.find('sensor.other.radar')
        radar_bp.set_attribute('horizontal_fov', '35')
        radar_bp.set_attribute('vertical_fov', '20')
        radar_bp.set_attribute('range', '70')
        radar = self.world.spawn_actor(
            radar_bp,
            carla.Transform(carla.Location(x=2.5, z=0.5)),
            attach_to=ego
        )

        return camera, lidar, radar

    def evaluate_fusion(self, fused_detections: list,
                        ground_truth_actors: list,
                        ego_location: carla.Location,
                        match_threshold_m: float = 2.0) -> dict:
        """
        Compare fused detections to CARLA ground truth actors.
        Returns precision, recall, and mean position error.
        """
        gt_positions = [
            np.array([a.get_location().x - ego_location.x,
                      a.get_location().y - ego_location.y])
            for a in ground_truth_actors
            if a.get_location().distance(ego_location) < 70.0
        ]

        matched_gt = set()
        matched_det = set()

        for det_idx, det in enumerate(fused_detections):
            det_pos = np.array([det['x'], det['y']])
            for gt_idx, gt_pos in enumerate(gt_positions):
                if gt_idx in matched_gt:
                    continue
                dist = np.linalg.norm(det_pos - gt_pos)
                if dist < match_threshold_m:
                    matched_gt.add(gt_idx)
                    matched_det.add(det_idx)
                    break

        tp = len(matched_gt)
        fp = len(fused_detections) - len(matched_det)
        fn = len(gt_positions) - tp

        precision = tp / (tp + fp) if (tp + fp) > 0 else 1.0
        recall = tp / (tp + fn) if (tp + fn) > 0 else 1.0

        return {
            'precision': precision,
            'recall': recall,
            'false_positive_count': fp,
            'false_negative_count': fn,
            'ground_truth_count': len(gt_positions)
        }

Edge Case Scenario Library

CARLA's ScenarioRunner supports parameterized scenario definitions. Build a library of edge cases that would be impractical to test on a real road:

# edge_cases.py — library of CARLA scenario parameters

EDGE_CASE_SCENARIOS = [
    {
        'id': 'EC-001',
        'name': 'Pedestrian behind sun glare',
        'weather': carla.WeatherParameters(
            sun_altitude_angle=5.0,    # Low sun angle
            sun_azimuth_angle=0.0,     # Directly in camera FOV
            cloudiness=0.0,
            fog_density=0.0
        ),
        'expected_detection_recall': 0.85,  # Relaxed for camera degradation
    },
    {
        'id': 'EC-002',
        'name': 'Heavy rain + reduced radar range',
        'weather': carla.WeatherParameters(
            precipitation=100.0,
            precipitation_deposits=80.0,
            wind_intensity=60.0
        ),
        'radar_range_override_m': 40.0,  # Rain attenuates radar
        'expected_detection_recall': 0.90,
    },
    {
        'id': 'EC-003',
        'name': 'LiDAR occlusion by large truck',
        'occluder': {'type': 'vehicle.carlamotors.carlacola', 'distance_m': 8.0},
        'target': {'type': 'walker.pedestrian.*', 'behind_occluder': True},
        'expected_detection_recall': 0.70,  # Camera must compensate
    },
    {
        'id': 'EC-004',
        'name': 'Ghost object from road surface reflection',
        'weather': carla.WeatherParameters(wetness=100.0),
        'expected_false_positive_rate': 0.02,  # Max 2% FP rate in wet conditions
    },
]

CI Integration for Simulation Tests

Running CARLA scenarios in CI requires headless mode and GPU access. Here's a GitHub Actions workflow using self-hosted runners with NVIDIA GPUs:

# .github/workflows/simulation-tests.yml
name: ADAS Simulation Tests

on:
  push:
    branches: [main, develop]
  schedule:
    - cron: '0 2 * * *'  # Nightly full suite

jobs:
  carla-scenarios:
    runs-on: [self-hosted, gpu, linux]
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v4

      - name: Start CARLA server (headless)
        run: |
          /opt/carla/CarlaUE4.sh \
            -RenderOffScreen \
            -quality-level=Low \
            -world-port=2000 \
            -benchmark -fps=20 &
          # Wait for server ready
          python3 -c "
          import carla, time
          for i in range(30):
              try:
                  c = carla.Client('localhost', 2000)
                  c.set_timeout(2.0)
                  c.get_world()
                  print('CARLA ready')
                  break
              except:
                  time.sleep(1)
          "

      - name: Run AEB scenarios
        run: |
          python3 tests/simulation/run_aeb_scenarios.py \
            --scenarios configs/aeb_scenarios.yaml \
            --output results/aeb_results.json \
            --fail-on-any-failure

      - name: Run sensor fusion edge cases
        run: |
          python3 tests/simulation/run_edge_cases.py \
            --scenarios edge_cases.py \
            --output results/edge_case_results.json

      - name: Generate metrics report
        run: |
          python3 tests/simulation/generate_report.py \
            --results results/*.json \
            --output simulation_report.html \
            --thresholds configs/pass_thresholds.yaml

      - name: Check pass/fail thresholds
        run: |
          python3 - <<'EOF'
          import json

          with open('results/aeb_results.json') as f:
              results = json.load(f)

          failures = [r for r in results if not r['passed']]
          if failures:
              for f in failures:
                  print(f"FAIL: {f['scenario_id']} - {f['reason']}")
              raise SystemExit(1)

          recall_values = [r['metrics']['recall'] for r in results]
          avg_recall = sum(recall_values) / len(recall_values)
          if avg_recall < 0.95:
              print(f"Mean recall {avg_recall:.3f} below 0.95 threshold")
              raise SystemExit(1)

          fp_rates = [r['metrics']['false_positive_rate'] for r in results]
          max_fp = max(fp_rates)
          if max_fp > 0.05:
              print(f"Max false positive rate {max_fp:.3f} exceeds 0.05 limit")
              raise SystemExit(1)

          print(f"All scenarios passed. Mean recall: {avg_recall:.3f}, Max FP rate: {max_fp:.4f}")
          EOF

      - name: Upload simulation artifacts
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: simulation-results
          path: |
            results/
            simulation_report.html

Key Metrics to Track

Track these metrics across scenario runs to detect regressions in your ADAS algorithms:

Metric Definition Target Measurement
True Positive Rate (Recall) GT objects detected / total GT objects > 95% Per scenario
False Positive Rate False detections / total detections < 2% Per scenario
AEB Trigger Latency Time from pedestrian visibility to braking < 300ms P99 across runs
False Brake Rate AEB triggers on safe scenarios 0 Count
Position Error Mean distance GT-to-fused object < 0.5m RMSE
Scenario Pass Rate Passed / total scenarios 100% Per commit

Track these over time with a tool like MLflow or simply store in a time-series database (InfluxDB works fine). A sudden drop in recall on EC-003 (LiDAR occlusion) after a sensor fusion algorithm change is immediately visible.

Simulation gives you reproducibility and scale that road testing never can. A scenario that took 6 hours of driving to capture once can be replayed 10,000 times overnight. Build the library, automate the metrics, and treat simulation regressions with the same urgency as unit test failures.

Read more

Start now free