AI Testing

How to Test Computer Vision Models: PyTorch, OpenCV, and YOLO

HelpMeTest

27 Aug 2026 — 8 min read

Computer vision models break in ways that unit tests don't catch. Your preprocessing pipeline silently changes a pixel value range, your bounding box coordinates are in the wrong format, or your model returns detections in a different order than your downstream code expects. None of these show up in accuracy metrics — they show up as wrong behavior in production.

This guide covers how to write tests for CV pipelines: unit testing preprocessing and postprocessing, integration testing model inference, regression testing YOLO detections, and testing OpenCV transforms.

What to Test in a CV Pipeline

A typical CV pipeline has three stages:

Input Image → Preprocessing → Model Inference → Postprocessing → Output

Each stage is independently testable:

Preprocessing — resize, normalize, format conversion, augmentation
Model inference — output tensor shape, value ranges, deterministic output
Postprocessing — bounding box decoding, NMS, coordinate scaling, class mapping

Most bugs live in preprocessing and postprocessing. Test those thoroughly; test model inference for smoke/regression.

Setting Up

pip install pytest pytest-cov numpy opencv-python torch torchvision ultralytics Pillow

Use pytest with fixtures for test images:

# conftest.py
import pytest
import numpy as np
from PIL import Image
import os

@pytest.fixture
def black_100x100():
    """Solid black 100×100 RGB image."""
    return np.zeros((100, 100, 3), dtype=np.uint8)

@pytest.fixture
def checkerboard_256():
    """256×256 checkerboard pattern."""
    img = np.zeros((256, 256, 3), dtype=np.uint8)
    for i in range(0, 256, 32):
        for j in range(0, 256, 32):
            if (i // 32 + j // 32) % 2 == 0:
                img[i:i+32, j:j+32] = 255
    return img

@pytest.fixture
def sample_rgb_image():
    """Synthetic image with known pixel values for testing normalization."""
    img = np.full((224, 224, 3), 128, dtype=np.uint8)
    img[0, 0] = [0, 0, 0]      # black pixel at origin
    img[0, 1] = [255, 255, 255]  # white pixel
    return img

Testing Preprocessing Pipelines

Preprocessing bugs are common and silent. Test each transform:

# tests/test_preprocessing.py
import numpy as np
import pytest
from src.preprocessing import preprocess_for_inference

class TestNormalization:
    def test_output_range_is_zero_to_one(self, sample_rgb_image):
        result = preprocess_for_inference(sample_rgb_image)
        assert result.min() >= 0.0, "Values below 0 after normalization"
        assert result.max() <= 1.0, "Values above 1 after normalization"

    def test_imagenet_normalization_shifts_values(self, sample_rgb_image):
        """ImageNet normalization: mean=[0.485,0.456,0.406], std=[0.229,0.224,0.225]"""
        result = preprocess_for_inference(sample_rgb_image, normalize="imagenet")
        # After ImageNet normalization, a pixel with value 128/255 ≈ 0.502
        # should land near 0.0 on each channel
        # This test catches if someone uses wrong mean/std values
        center_pixel = result[:, 112, 112]  # CHW format
        assert abs(center_pixel[0]) < 0.2, f"Red channel unexpectedly far from 0: {center_pixel[0]}"

    def test_output_dtype_is_float32(self, sample_rgb_image):
        import torch
        result = preprocess_for_inference(sample_rgb_image)
        assert result.dtype == torch.float32

    def test_black_pixel_maps_to_expected_value(self, sample_rgb_image):
        result = preprocess_for_inference(sample_rgb_image)
        # Black pixel (0/255 = 0.0), after ImageNet normalization:
        # (0.0 - 0.485) / 0.229 ≈ -2.118 for red channel
        black_pixel = result[:, 0, 0]
        assert black_pixel[0] < -1.5, f"Black pixel red channel: {black_pixel[0]}"


class TestResize:
    def test_output_is_target_size(self, black_100x100):
        from src.preprocessing import resize_image
        result = resize_image(black_100x100, target_size=(224, 224))
        assert result.shape[:2] == (224, 224), f"Expected (224, 224), got {result.shape[:2]}"

    def test_non_square_input_pads_correctly(self):
        from src.preprocessing import resize_with_padding
        # Wide image: 100×50
        wide_image = np.zeros((50, 100, 3), dtype=np.uint8)
        wide_image[:, :] = [100, 150, 200]  # distinctive color

        result = resize_with_padding(wide_image, target_size=64)

        assert result.shape == (64, 64, 3)
        # Center should have the original image content, not padding
        assert not np.all(result[32, 32] == 0)

    def test_resize_preserves_channel_order(self):
        from src.preprocessing import resize_image
        # Create image with distinct R, G, B channels
        img = np.zeros((100, 100, 3), dtype=np.uint8)
        img[:, :, 0] = 100  # R
        img[:, :, 1] = 150  # G
        img[:, :, 2] = 200  # B

        result = resize_image(img, target_size=(50, 50))

        assert result[25, 25, 0] == pytest.approx(100, abs=5)
        assert result[25, 25, 1] == pytest.approx(150, abs=5)
        assert result[25, 25, 2] == pytest.approx(200, abs=5)


class TestFormatConversion:
    def test_bgr_to_rgb_swaps_channels(self, black_100x100):
        from src.preprocessing import bgr_to_rgb
        # OpenCV loads in BGR; models expect RGB
        bgr_image = np.zeros((100, 100, 3), dtype=np.uint8)
        bgr_image[:, :, 0] = 255  # Blue channel in BGR

        rgb_image = bgr_to_rgb(bgr_image)

        # Blue (BGR[0]) should now be in RGB[2]
        assert rgb_image[0, 0, 2] == 255
        assert rgb_image[0, 0, 0] == 0

    def test_hwc_to_chw_transpose(self, sample_rgb_image):
        from src.preprocessing import hwc_to_chw
        # Input: (H, W, C) → Output: (C, H, W)
        result = hwc_to_chw(sample_rgb_image)
        assert result.shape == (3, 224, 224), f"Expected (3, 224, 224), got {result.shape}"

Testing Model Inference

Test model outputs for shape correctness, value ranges, and determinism:

# tests/test_model.py
import torch
import pytest
from src.model import load_classifier

@pytest.fixture(scope="module")
def model():
    return load_classifier("resnet18", pretrained=False)

class TestModelInference:
    def test_output_shape_matches_num_classes(self, model):
        input_tensor = torch.randn(1, 3, 224, 224)
        with torch.no_grad():
            output = model(input_tensor)
        # ResNet18 with 1000 classes
        assert output.shape == (1, 1000), f"Unexpected output shape: {output.shape}"

    def test_output_is_not_nan_or_inf(self, model):
        input_tensor = torch.randn(4, 3, 224, 224)
        with torch.no_grad():
            output = model(input_tensor)
        assert not torch.isnan(output).any(), "NaN in model output"
        assert not torch.isinf(output).any(), "Inf in model output"

    def test_softmax_probabilities_sum_to_one(self, model):
        import torch.nn.functional as F
        input_tensor = torch.randn(1, 3, 224, 224)
        with torch.no_grad():
            logits = model(input_tensor)
        probs = F.softmax(logits, dim=1)
        assert abs(probs.sum().item() - 1.0) < 1e-5

    def test_deterministic_output_in_eval_mode(self, model):
        model.eval()
        input_tensor = torch.randn(1, 3, 224, 224)
        with torch.no_grad():
            out1 = model(input_tensor)
            out2 = model(input_tensor)
        assert torch.allclose(out1, out2), "Non-deterministic output in eval mode"

    def test_batch_output_matches_single_inference(self, model):
        """Batch processing must match individual processing."""
        model.eval()
        inputs = torch.randn(3, 3, 224, 224)
        with torch.no_grad():
            batch_out = model(inputs)
            single_outs = torch.stack([model(inputs[i:i+1]) for i in range(3)])
        # Outputs should be identical whether processed as batch or individually
        assert torch.allclose(batch_out, single_outs.squeeze(1), atol=1e-5)

Testing YOLO Object Detection

YOLO detection has unique postprocessing to test — bounding boxes, confidence scores, NMS, coordinate formats:

# tests/test_yolo.py
import numpy as np
import pytest
from src.detection import YOLODetector, Detection

class TestYOLODetector:
    @pytest.fixture(scope="class")
    def detector(self):
        return YOLODetector(model_path="yolov8n.pt", conf_threshold=0.5)

    def test_detection_output_format(self, detector, checkerboard_256):
        detections = detector.detect(checkerboard_256)
        assert isinstance(detections, list)
        for det in detections:
            assert isinstance(det, Detection)
            assert hasattr(det, "bbox")
            assert hasattr(det, "confidence")
            assert hasattr(det, "class_id")
            assert hasattr(det, "class_name")

    def test_bounding_box_coordinates_within_image_bounds(self, detector):
        """BBoxes must not extend outside the image."""
        import cv2
        img = np.zeros((480, 640, 3), dtype=np.uint8)
        # Draw a rectangle that looks like an object
        cv2.rectangle(img, (100, 100), (300, 300), (255, 255, 255), -1)

        detections = detector.detect(img)
        h, w = img.shape[:2]

        for det in detections:
            x1, y1, x2, y2 = det.bbox
            assert x1 >= 0, f"x1={x1} outside image"
            assert y1 >= 0, f"y1={y1} outside image"
            assert x2 <= w, f"x2={x2} outside image width {w}"
            assert y2 <= h, f"y2={y2} outside image height {h}"
            assert x2 > x1, f"x2={x2} not greater than x1={x1}"
            assert y2 > y1, f"y2={y2} not greater than y1={y1}"

    def test_confidence_scores_in_valid_range(self, detector, checkerboard_256):
        detections = detector.detect(checkerboard_256)
        for det in detections:
            assert 0.0 <= det.confidence <= 1.0, \
                f"Confidence {det.confidence} out of range"
            assert det.confidence >= 0.5, \
                f"Detection below threshold: {det.confidence}"

    def test_class_names_are_valid_coco_labels(self, detector, checkerboard_256):
        COCO_CLASSES = {
            "person", "bicycle", "car", "motorcycle", "airplane",
            "bus", "train", "truck", "boat", "cat", "dog",
            # ... add full COCO class list
        }
        detections = detector.detect(checkerboard_256)
        for det in detections:
            assert det.class_name in COCO_CLASSES or True, \
                f"Unknown class: {det.class_name}"  # permissive for non-COCO models


class TestNMSPostprocessing:
    """Test Non-Maximum Suppression removes duplicate detections."""

    def test_nms_removes_overlapping_boxes(self):
        from src.detection import apply_nms

        # Two heavily overlapping boxes for the same class
        boxes = np.array([
            [100, 100, 200, 200],  # Box A
            [105, 105, 205, 205],  # Box B (90%+ IoU with A)
        ], dtype=np.float32)
        scores = np.array([0.9, 0.8])
        class_ids = np.array([0, 0])

        result = apply_nms(boxes, scores, class_ids, iou_threshold=0.5)

        assert len(result) == 1, f"NMS should keep 1 box, kept {len(result)}"
        # Should keep the higher-confidence box
        assert result[0].confidence == pytest.approx(0.9)

    def test_nms_keeps_non_overlapping_boxes(self):
        from src.detection import apply_nms

        # Two boxes far apart
        boxes = np.array([
            [10, 10, 50, 50],    # Top-left
            [200, 200, 250, 250], # Bottom-right
        ], dtype=np.float32)
        scores = np.array([0.9, 0.85])
        class_ids = np.array([0, 0])

        result = apply_nms(boxes, scores, class_ids, iou_threshold=0.5)

        assert len(result) == 2, "NMS should keep both non-overlapping boxes"

Testing OpenCV Image Transforms

OpenCV operations have subtle gotchas — channel order, data types, in-place vs. copy:

# tests/test_opencv_transforms.py
import cv2
import numpy as np
import pytest

class TestOpenCVTransforms:
    def test_threshold_binarizes_correctly(self):
        from src.transforms import adaptive_threshold
        # Gradient image: left half dark, right half bright
        img = np.zeros((100, 100), dtype=np.uint8)
        img[:, 50:] = 200  # right half bright

        result = adaptive_threshold(img, threshold=100)

        assert result.dtype == np.uint8
        assert np.all(result[:, :50] == 0), "Dark region should be black"
        assert np.all(result[:, 50:] == 255), "Bright region should be white"

    def test_canny_edge_detection_returns_single_channel(self):
        from src.transforms import detect_edges
        color_img = np.zeros((100, 100, 3), dtype=np.uint8)
        cv2.rectangle(color_img, (20, 20), (80, 80), (255, 255, 255), 2)

        edges = detect_edges(color_img)

        assert len(edges.shape) == 2, f"Expected 2D, got shape {edges.shape}"
        assert edges.dtype == np.uint8

    def test_morphological_dilation_expands_region(self):
        from src.transforms import dilate
        # Single white pixel
        img = np.zeros((50, 50), dtype=np.uint8)
        img[25, 25] = 255

        result = dilate(img, kernel_size=5)

        # After 5×5 dilation, should have a 5×5 region (approx)
        white_pixels = np.sum(result > 0)
        assert white_pixels >= 25, f"Dilation too small: {white_pixels} pixels"
        assert white_pixels <= 100, f"Dilation too large: {white_pixels} pixels"

    def test_gaussian_blur_reduces_noise(self):
        from src.transforms import apply_gaussian_blur
        # Add salt-and-pepper noise
        noisy = np.zeros((100, 100, 3), dtype=np.uint8)
        noisy[::5, ::5] = 255  # periodic white dots

        blurred = apply_gaussian_blur(noisy, kernel_size=5)

        # Max pixel value should be reduced by blur
        assert blurred.max() < noisy.max(), "Blur should reduce peak values"
        # Mean should be similar
        assert abs(int(blurred.mean()) - int(noisy.mean())) < 20

Regression Testing with Golden Outputs

For detection regression, save expected outputs and compare:

# tests/test_regression.py
import json
import numpy as np
import pytest
from pathlib import Path

GOLDEN_DIR = Path("tests/golden")

def test_yolo_detection_regression():
    from src.detection import YOLODetector
    import cv2

    detector = YOLODetector("yolov8n.pt")
    
    # Use a fixed test image committed to the repo
    img = cv2.imread("tests/fixtures/test_street_scene.jpg")
    detections = detector.detect(img)

    golden_path = GOLDEN_DIR / "test_street_scene_detections.json"

    if not golden_path.exists():
        # First run: save golden output
        GOLDEN_DIR.mkdir(exist_ok=True)
        golden_path.write_text(
            json.dumps(
                [
                    {
                        "class_name": d.class_name,
                        "bbox": [round(v, 1) for v in d.bbox],
                        "confidence_floor": round(d.confidence, 1),  # saves floor, not exact value
                    }
                    for d in sorted(detections, key=lambda d: d.bbox[0])
                ],
                indent=2,
            )
        )
        pytest.skip("Golden output created — run again to verify")

    golden = json.loads(golden_path.read_text())
    detected_classes = sorted([d.class_name for d in detections])
    expected_classes = sorted([g["class_name"] for g in golden])

    assert detected_classes == expected_classes, (
        f"Detection classes changed.\n"
        f"Expected: {expected_classes}\n"
        f"Got: {detected_classes}"
    )

CI Configuration

# .github/workflows/cv-tests.yml
- name: Run CV tests
  run: |
    pip install -r requirements-test.txt
    pytest tests/ -v --cov=src --cov-report=xml
  env:
    # Disable GPU in CI for reproducibility
    CUDA_VISIBLE_DEVICES: ""

Force CPU inference to avoid GPU-related flakiness in CI:

# conftest.py
import os
os.environ.setdefault("CUDA_VISIBLE_DEVICES", "")

import torch
torch.set_default_device("cpu")

The Most Common CV Testing Bugs

BGR/RGB channel flip — OpenCV loads BGR, PyTorch/YOLO expect RGB. Test channel order explicitly.
Float range mismatch — model expects [0, 1], you pass [0, 255]. Test normalization output range.
HWC vs. CHW format — NumPy is HWC, PyTorch is CHW. Test tensor shape after conversion.
Bounding box format — [x1, y1, x2, y2] vs [x, y, w, h] vs normalized [0, 1]. Test all coordinate conversions.
Batch dimension — model expects (B, C, H, W), you pass (C, H, W). Test batch dimension insertion.

These five bugs cause 80% of CV pipeline failures. If you test for them explicitly, you catch them before production.

How to Test Computer Vision Models: PyTorch, OpenCV, and YOLO

HelpMeTest

What to Test in a CV Pipeline

Setting Up

Testing Preprocessing Pipelines

Testing Model Inference

Testing YOLO Object Detection

Testing OpenCV Image Transforms

Regression Testing with Golden Outputs

CI Configuration

The Most Common CV Testing Bugs

Read more

Test Result Reporting and Failing Fast in CI Pipelines

SOC 2 Evidence Collection Automation: Stop Emailing Screenshots to Your Auditor

PCI-DSS 4.0 Penetration Testing: Scoping, Methodology, and What Assessors Actually Check

Data Pipeline CI/CD Testing: Automate Quality Gates