AI Testing
Model regression testing is the practice of verifying that an LLM upgrade, prompt change, or fine-tune didn't make things worse. It sounds obvious. Most teams don't do it. The typical pattern: team upgrades from GPT-4 to GPT-4o to save costs. Everything seems fine
AI Testing
Hallucination is when an LLM presents false information as true. The model doesn't know it's lying — it's pattern-matching to produce plausible-sounding text, and sometimes plausible diverges from factual. For most applications, hallucinations aren't a curiosity — they're a production
Testing
Agentic AI evaluation goes beyond single-turn accuracy. Measure task completion rate, tool-call precision/recall, step efficiency, and trajectory quality. A 90% single-turn accuracy agent can still fail 40% of multi-step tasks due to error propagation. Evaluate the full trajectory, not just the final answer. Single-turn
Testing
AI chatbots fail in ways that traditional software doesn't. They misunderstand intent, lose context across turns, make up information, break character, and occasionally say things that would horrify your legal team. Testing them requires a different mental model from testing deterministic systems — you're testing behaviors under
Testing
LLMs hallucinate. They generate plausible-sounding text that is factually wrong, cite papers that don't exist, quote statistics that were never measured, and confidently describe events that never happened. If your product uses an LLM to generate content that users trust — customer support answers, medical information, legal summaries,
Testing
ML models degrade in ways that traditional software doesn't. Code doesn't spontaneously get worse — but a model trained on March data can silently degrade by June as the distribution of real-world inputs shifts. A retrained model can score higher on aggregate metrics while regressing on
Testing
A/B testing AI features is harder than testing traditional software. Changing a button color produces a measurable, binary outcome — users click or they don't. Changing an LLM prompt produces different text outputs, and measuring which output is "better" requires defining what better even means for
AI Testing
Adding AI features to your SaaS is easy. Shipping them with the same quality bar as the rest of your product is hard. A traditional feature has deterministic behavior — a button either submits a form or it doesn't. An AI feature has probabilistic behavior — the same input produces
AI Testing
Weights & Biases built its reputation on ML experiment tracking — recording every hyperparameter, metric, and artifact from model training runs. Weave extends that discipline to LLM applications, where the "experiment" isn't a training run but a prompt, a retrieval config, or a pipeline change. If you&
AI Testing
Every time you change a prompt, update a model version, modify your retrieval pipeline, or adjust a system instruction, you've potentially introduced a regression. Unlike traditional software regressions — where a function returns the wrong value and a unit test catches it — LLM regressions are subtle. The output format
AI Testing
Most LLM applications fail at the infrastructure layer, not the model layer. Your prompt is fine. Your retrieval is fine. But when OpenAI's API goes down at 2 AM, or when your token budget runs out mid-sprint, the application breaks entirely. Portkey is an AI gateway that
AI Testing
LLM applications fail in ways that traditional monitoring misses entirely. A REST API either returns 200 or it doesn't. An LLM can return 200 and still produce hallucinated facts, broken reasoning chains, or responses that drift from your prompt instructions over time. Arize Phoenix exists to make that