Testing AI Features Without Flakiness: A Practical Framework
Flaky tests are a plague in any test suite — tests that sometimes pass and sometimes fail without any code change. AI features are especially prone to producing flaky tests because the underlying behavior is non-deterministic. The same input to an LLM produces different output every time.
But flaky tests are a choice, not a requirement. You can write reliable tests for AI features. The key is testing the right things — the deterministic parts of your system, not the non-deterministic parts.
Why AI Test Flakiness Happens
Before fixing flakiness, understand where it comes from.
Asserting on generated content. The most common cause: a test that asserts response.contains("specific phrase") when the phrase comes from an LLM. The LLM sometimes includes that phrase, sometimes doesn't. The test is inherently flaky.
Variable latency. AI API calls take 1-30 seconds depending on model, input size, and provider load. A test with a 5-second timeout will sometimes pass (fast response) and sometimes fail (slow response). Flaky.
Model version drift. If you use a "latest" model alias instead of pinning to a specific version, the model can update between test runs. Yesterday's test passed with one model behavior; today it fails with updated behavior. Technically not flaky (behavior changed), but it presents as flakiness.
Rate limits. If your test suite calls the API at volume, rate limits cause tests to fail intermittently. The failure rate depends on other traffic, time of day, and current API tier.
Floating reference data. If your test creates data, runs an AI operation on it, then asserts on the result — and the test doesn't clean up properly — stale data from previous runs can affect results.
Each of these has a specific fix.
The Right Things to Test in AI Features
The solution to AI test flakiness is testing the deterministic parts of your system, not the non-deterministic parts.
Deterministic parts of an AI feature:
- Input preprocessing (cleaning, chunking, formatting)
- Output parsing (extracting structured data from LLM responses)
- Error handling (how the system responds to LLM failures)
- UI state management (loading states, error states, success states)
- Caching and rate limiting logic
- Cost controls and token counting
Non-deterministic parts:
- The specific content of LLM responses
- The exact phrasing of generated text
- The specific examples chosen by the model
Test the deterministic parts with standard unit tests. For the non-deterministic parts, test properties and behaviors rather than specific content.
Strategies for Non-Flaky AI Tests
Strategy 1: Pin the model and use temperature=0
For unit and integration tests, use a specific model version (gpt-4o-2024-11-20, not gpt-4o) and set temperature=0. This makes responses deterministic enough for reliable testing.
def get_test_llm_client():
return openai.OpenAI()
def test_extraction_returns_product_name():
response = extract_product_info(
"Buy the Blue Widget today for $29.99",
model="gpt-4o-2024-11-20",
temperature=0
)
# Deterministic with temperature=0 and pinned model
assert response['product_name'] == 'Blue Widget'
assert response['price'] == 29.99Caveat: temperature=0 with the same model version is approximately deterministic, not perfectly deterministic. For highly sensitive assertions, use mocks instead.
Strategy 2: Test properties, not content
When temperature=0 is insufficient (or you're using production settings), assert on properties of the output rather than specific content.
Flaky assertion:
assert "The main character learns a valuable lesson" in summaryReliable assertion:
assert len(summary) > 50 # summary exists
assert len(summary) < len(original) * 0.5 # actually shorter
assert summary.strip() != '' # not emptyFlaky assertion:
assert response['tone'] == 'friendly'Reliable assertion:
assert response['tone'] in ['friendly', 'professional', 'casual']
assert 'tone' in response # key existsStrategy 3: Mock for unit tests, real API for integration tests
Unit tests should never call the real LLM API. Use fixtures with realistic responses:
@pytest.fixture
def mock_openai(monkeypatch):
def mock_create(**kwargs):
# Return a realistic response for the input
return MockResponse(
content='{"product_name": "Blue Widget", "price": 29.99}',
tokens=150
)
monkeypatch.setattr('openai.ChatCompletion.create', mock_create)
def test_extraction_logic(mock_openai):
result = extract_product_info("Buy the Blue Widget today for $29.99")
assert result['product_name'] == 'Blue Widget'Save real API calls for integration tests. Run integration tests less frequently (nightly or on release, not on every commit).
Strategy 4: Fix the latency timeout problem
Don't use hardcoded timeouts for AI API calls in tests. Use generous timeouts and mark slow tests:
@pytest.mark.slow
@pytest.mark.timeout(120) # 2 minutes — LLM calls can be slow
def test_document_processing_end_to_end():
result = process_document("tests/fixtures/sample.pdf")
assert result.is_processed
assert result.summary is not NoneFor behavioral tests that run in a browser, use explicit waits rather than sleep:
Click Generate Summary
Wait Until Element Is Visible id:summary-content timeout=60s
Element Should Contain Text id:summary-content ${EMPTY} NOTThe 60-second wait is generous. It doesn't make the test slow when LLM is fast — it just allows for slowness. This eliminates the timeout-based flakiness.
Strategy 5: Isolate API rate limit failures
Rate limit errors shouldn't be flaky test failures — they should be explicit skips with clear reasons:
@pytest.mark.api_integration
def test_batch_processing():
try:
results = process_batch(test_items)
assert len(results) == len(test_items)
except RateLimitError:
pytest.skip("Rate limit hit — retry later")Better: use a separate test LLM account with dedicated quota for your test suite.
Behavioral Testing for AI Features
Behavioral tests that run in a real browser are the most flakiness-prone AI tests if done wrong, and the most reliable if done right.
The right approach:
- Assert on UI state, not content
- Use generous timeouts
- Test that the feature responds, not what it says
Flaky behavioral test:
Go to https://myapp.com/summarize
Enter test article
Click Summarize
Wait 5 seconds
Verify page contains "In summary,"Reliable behavioral test:
Go to https://myapp.com/summarize
Enter test article
Click Summarize
Wait for "Generating summary..." to appear
Wait for "Generating summary..." to disappear (timeout: 90 seconds)
Verify element "#summary-output" is visible
Verify element "#summary-output" is not empty
Verify no error message is visibleThe reliable version:
- Waits for explicit UI signals, not fixed time
- Asserts on presence and non-emptiness, not specific content
- Has a generous timeout that handles slow responses
HelpMeTest runs these behavioral tests continuously against production — every 5 minutes. Non-flaky behavioral tests are critical here because a flaky test would trigger false alerts.
The Flakiness Audit
If you already have a flaky AI test suite, here's how to audit it:
- Categorize failures. Run the suite 5 times. Which tests fail sometimes? What's the failure rate for each?
- Find the source. For each flaky test, identify which of the causes applies: content assertion, timeout, rate limit, model drift.
- Fix by category:
- Content assertion → replace with property assertion
- Timeout → add generous explicit waits
- Rate limit → use mocks in unit tests, separate quota for integration tests
- Model drift → pin the model version
- Validate the fix. Run the fixed test 10 times. If it passes every time, the flakiness is fixed.
The Stable Test Suite
A stable AI test suite:
- Unit tests: mock LLM calls, test transformation logic, never flaky
- Integration tests: real API, pinned model, temperature=0, run nightly
- Behavioral tests: real browser, property assertions, generous timeouts, run continuously in production
This structure gives you fast feedback (unit tests on every commit), real integration verification (integration tests nightly), and continuous production monitoring (behavioral tests always on).
Flaky tests are worse than no tests. They train your team to ignore failures. Build the stable foundation first.
HelpMeTest provides reliable behavioral monitoring for AI features — tests that run continuously without crying wolf.