MLOps
Weights & Biases (W&B) tracks ML experiments — but it also provides the infrastructure to assert training quality, detect regressions, and gate deployments on metric thresholds. This post covers how to use W&B as a testing tool, not just a logging tool. W&B as a
MLOps
OpenAI Evals is a framework for evaluating large language model outputs. It lets you define test cases (called "evals") that measure whether a model produces correct, consistent, or appropriately formatted responses — and run them as automated checks in your development pipeline. What Evals Test Evals measure properties of
MLOps
Evidently AI is an open-source framework for evaluating and monitoring machine learning models. It generates reports and test suites that check data drift, model quality, and dataset statistics — making it a core tool for MLOps testing pipelines. What Evidently Tests Evidently organizes checks into three categories: * Data quality — missing
MLOps
Seldon Core and KServe deploy machine learning models as Kubernetes-native inference services. Testing these deployments means verifying the inference API, the Kubernetes resource state, canary rollouts, and model-level correctness — before and after going to production. What to Test Layer What to verify Kubernetes resources InferenceService is Ready, replicas
MLOps
BentoML packages machine learning models as production-ready API services. Testing BentoML services means verifying the model logic, the API layer, and the deployment — each at a different level. BentoML Service Structure A BentoML service wraps a model and exposes it as an HTTP API: # service.py import bentoml import
Testing
Great Expectations (GX) is the de facto standard for data quality validation in ML pipelines. It lets you define data expectations as code, validate datasets before training, and generate visual data quality reports. This guide covers setup, writing expectations, checkpoints, and pipeline integration. Why Data Validation Matters for ML The
Testing
ML models degrade silently as the world changes. Data drift tests verify that incoming features match training distribution. Concept drift tests detect when the relationship between features and labels has changed. This guide covers both with Evidently, Alibi Detect, and CI integration. The Drift Problem You deploy a credit risk
Testing
MLflow provides the infrastructure for systematic ML model testing: experiment tracking for comparison, model validation for quality gates, and a model registry for staging promotion. This guide covers all three with practical test patterns for CI/ML pipelines. Why ML Models Need Tests Software tests assert that code does what
Machine Learning
Feature stores solve the problem of feature reuse and training/serving consistency in ML systems. They also introduce a new category of bugs: features that compute correctly in isolation but produce wrong values when served at inference time due to point-in-time leakage, online/offline discrepancies, or stale data.
Machine Learning
A model you deploy today will degrade over time. The world changes — user behavior shifts, economic conditions fluctuate, upstream data systems evolve. The model doesn't know. Without drift monitoring, you find out when a user complains or a business metric tanks. Drift detection is the practice of monitoring
Machine Learning
Offline evaluation tells you a model performs better on held-out data. Production A/B testing tells you it actually improves the business metric you care about. These are different questions, and sometimes they have different answers. A new recommendation model might have higher offline AUC but lower click-through