Groq API Testing: Streaming, Latency Benchmarks, and Error Handling
Groq stands apart in the LLM landscape for one reason: speed. The Language Processing Unit (LPU) architecture delivers token generation that's often 10-20x faster than GPU-based inference. But raw speed creates new testing challenges — your test suite needs to validate not just correctness but latency characteristics, streaming behavior, and the specific error modes that come with an infrastructure built for throughput.
This guide covers everything you need to test Groq API integrations in production-grade Python applications.
Why Groq Testing Is Unique
Testing Groq differs from testing other LLM providers because:
- Latency is a feature: Users choose Groq for speed. A correct but slow response is a regression.
- Streaming is the default path: Most Groq use cases use streaming; testing non-streaming is not enough.
- Context limits are strict: Groq's models have specific context windows. Tests need to exercise boundary conditions.
- Rate limits hit differently: At 30 req/min for free tier, rate limit errors happen in tests if you're not careful.
Setting Up Groq Test Environment
pip install groq pytest pytest-asyncio httpxconftest.py:
import pytest
from unittest.mock import MagicMock, patch
from groq import Groq, AsyncGroq
@pytest.fixture
def groq_client():
"""Groq client configured for testing."""
return Groq(api_key="gsk_test_key_never_real")
@pytest.fixture
def async_groq_client():
"""Async Groq client for async tests."""
return AsyncGroq(api_key="gsk_test_key_never_real")
@pytest.fixture
def mock_groq_response():
"""Standard mock Groq completion response."""
response = MagicMock()
response.id = "chatcmpl-test-123"
response.model = "llama3-8b-8192"
response.choices = [MagicMock()]
response.choices[0].index = 0
response.choices[0].message.role = "assistant"
response.choices[0].message.content = "Test response content"
response.choices[0].finish_reason = "stop"
response.usage.prompt_tokens = 15
response.usage.completion_tokens = 25
response.usage.total_tokens = 40
response.x_groq = MagicMock()
response.x_groq.usage.queue_time = 0.003
response.x_groq.usage.prompt_time = 0.012
response.x_groq.usage.completion_time = 0.025
response.x_groq.usage.total_time = 0.040
return responseTesting Groq Completion Calls
import pytest
from unittest.mock import patch, MagicMock
from groq import Groq
def test_basic_chat_completion(groq_client, mock_groq_response):
"""Basic completion call returns expected structure."""
with patch.object(groq_client.chat.completions, "create", return_value=mock_groq_response):
response = groq_client.chat.completions.create(
model="llama3-8b-8192",
messages=[{"role": "user", "content": "Hello"}]
)
assert response.choices[0].message.content == "Test response content"
assert response.choices[0].message.role == "assistant"
assert response.usage.total_tokens == 40
def test_model_selection():
"""Verify correct model is passed to the API."""
client = Groq(api_key="test-key")
target_model = "mixtral-8x7b-32768"
with patch.object(client.chat.completions, "create") as mock_create:
mock_create.return_value = MagicMock(
choices=[MagicMock(message=MagicMock(content="ok", role="assistant"))]
)
client.chat.completions.create(
model=target_model,
messages=[{"role": "user", "content": "test"}]
)
call_kwargs = mock_create.call_args.kwargs
assert call_kwargs["model"] == target_model
def test_system_prompt_included():
"""System prompts should be passed as the first message."""
client = Groq(api_key="test-key")
with patch.object(client.chat.completions, "create") as mock_create:
mock_create.return_value = MagicMock(
choices=[MagicMock(message=MagicMock(content="response", role="assistant"))]
)
client.chat.completions.create(
model="llama3-8b-8192",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What time is it?"}
]
)
messages = mock_create.call_args.kwargs["messages"]
assert messages[0]["role"] == "system"
assert "helpful assistant" in messages[0]["content"]
def test_temperature_and_max_tokens_passed():
"""Generation parameters should be forwarded correctly."""
client = Groq(api_key="test-key")
with patch.object(client.chat.completions, "create") as mock_create:
mock_create.return_value = MagicMock(
choices=[MagicMock(message=MagicMock(content="ok", role="assistant"))]
)
client.chat.completions.create(
model="llama3-8b-8192",
messages=[{"role": "user", "content": "test"}],
temperature=0.2,
max_tokens=500
)
kwargs = mock_create.call_args.kwargs
assert kwargs["temperature"] == 0.2
assert kwargs["max_tokens"] == 500Testing Groq Streaming Responses
Streaming is where Groq shines — and where tests get complex:
import time
from typing import Iterator
def make_mock_stream(chunks: list[str]) -> Iterator:
"""Create a mock streaming response."""
for i, text in enumerate(chunks):
chunk = MagicMock()
chunk.id = "chatcmpl-test"
chunk.model = "llama3-8b-8192"
chunk.choices = [MagicMock()]
chunk.choices[0].index = 0
chunk.choices[0].delta.role = "assistant" if i == 0 else None
chunk.choices[0].delta.content = text
chunk.choices[0].finish_reason = None if i < len(chunks) - 1 else "stop"
yield chunk
def test_streaming_collects_all_chunks(groq_client):
"""All stream chunks should be received and assembled."""
expected_tokens = ["Hello", ", ", "world", "!"]
with patch.object(groq_client.chat.completions, "create",
return_value=make_mock_stream(expected_tokens)):
stream = groq_client.chat.completions.create(
model="llama3-8b-8192",
messages=[{"role": "user", "content": "Say hello"}],
stream=True
)
collected = ""
for chunk in stream:
if chunk.choices[0].delta.content:
collected += chunk.choices[0].delta.content
assert collected == "Hello, world!"
def test_streaming_finish_reason():
"""Stream should end with finish_reason='stop' on normal completion."""
client = Groq(api_key="test-key")
chunks = ["Final", " chunk"]
with patch.object(client.chat.completions, "create",
return_value=make_mock_stream(chunks)):
stream = client.chat.completions.create(
model="llama3-8b-8192",
messages=[{"role": "user", "content": "test"}],
stream=True
)
last_chunk = None
for chunk in stream:
last_chunk = chunk
assert last_chunk is not None
assert last_chunk.choices[0].finish_reason == "stop"
@pytest.mark.asyncio
async def test_async_streaming(async_groq_client):
"""Async streaming should work with async for loops."""
async def async_stream():
for chunk in make_mock_stream(["async ", "response"]):
yield chunk
with patch.object(async_groq_client.chat.completions, "create",
return_value=async_stream()):
collected = ""
async with async_groq_client.chat.completions.create(
model="llama3-8b-8192",
messages=[{"role": "user", "content": "test"}],
stream=True
) as stream:
async for chunk in stream:
if chunk.choices[0].delta.content:
collected += chunk.choices[0].delta.content
assert collected == "async response"
def test_streaming_with_first_token_latency():
"""Measure time-to-first-token for streaming responses."""
client = Groq(api_key="test-key")
def slow_stream():
time.sleep(0.01) # Simulate network latency
yield from make_mock_stream(["First", " token"])
with patch.object(client.chat.completions, "create", return_value=slow_stream()):
start = time.monotonic()
stream = client.chat.completions.create(
model="llama3-8b-8192",
messages=[{"role": "user", "content": "test"}],
stream=True
)
first_token_time = None
for chunk in stream:
if chunk.choices[0].delta.content and first_token_time is None:
first_token_time = time.monotonic() - start
assert first_token_time is not None
# In production, you'd assert first_token_time < your_sla_thresholdLatency Benchmarking
Groq's selling point is speed. Here's how to benchmark it properly:
import statistics
import time
import pytest
class LatencyBenchmark:
"""Measure and assert latency characteristics."""
def __init__(self):
self.measurements = []
def record(self, duration_ms: float):
self.measurements.append(duration_ms)
@property
def p50(self) -> float:
return statistics.median(self.measurements)
@property
def p95(self) -> float:
sorted_m = sorted(self.measurements)
idx = int(len(sorted_m) * 0.95)
return sorted_m[min(idx, len(sorted_m) - 1)]
@property
def p99(self) -> float:
sorted_m = sorted(self.measurements)
idx = int(len(sorted_m) * 0.99)
return sorted_m[min(idx, len(sorted_m) - 1)]
def test_latency_benchmark_structure():
"""Verify benchmark captures statistics correctly."""
bench = LatencyBenchmark()
# Simulate 10 measurements
for ms in [45, 52, 48, 61, 43, 49, 55, 47, 50, 200]: # 200ms outlier
bench.record(ms)
assert bench.p50 < 60 # Median well under 60ms
assert bench.p95 < 150 # 95th percentile under 150ms
assert bench.p99 <= 200 # 99th under 200ms (our outlier)
@pytest.mark.benchmark
def test_groq_tokens_per_second(groq_client):
"""Calculate tokens per second from Groq response metadata."""
mock_response = MagicMock()
mock_response.choices = [MagicMock()]
mock_response.choices[0].message.content = "Test response " * 50
mock_response.usage.completion_tokens = 100
mock_response.x_groq = MagicMock()
mock_response.x_groq.usage.completion_time = 0.5 # 0.5 seconds
with patch.object(groq_client.chat.completions, "create", return_value=mock_response):
response = groq_client.chat.completions.create(
model="llama3-8b-8192",
messages=[{"role": "user", "content": "Write a short paragraph."}]
)
completion_tokens = response.usage.completion_tokens
completion_time = response.x_groq.usage.completion_time
tps = completion_tokens / completion_time
# Groq typically delivers 200-800 TPS depending on model
# This asserts the calculation is reasonable
assert tps > 0
assert tps < 2000 # Sanity upper boundTesting Error Handling
Groq-specific errors need explicit test coverage:
from groq import (
AuthenticationError,
RateLimitError,
APIStatusError,
APIConnectionError,
BadRequestError
)
import groq
def test_authentication_error_raised():
"""Invalid API key should raise AuthenticationError."""
client = Groq(api_key="invalid-key")
with patch.object(client.chat.completions, "create",
side_effect=AuthenticationError(
message="Invalid API Key",
response=MagicMock(status_code=401),
body={"error": {"message": "Invalid API Key"}}
)):
with pytest.raises(AuthenticationError) as exc_info:
client.chat.completions.create(
model="llama3-8b-8192",
messages=[{"role": "user", "content": "test"}]
)
assert "Invalid API Key" in str(exc_info.value)
def test_rate_limit_error_handling():
"""Rate limit errors should be caught and handled gracefully."""
client = Groq(api_key="test-key")
rate_limit_error = RateLimitError(
message="Rate limit exceeded for requests per minute",
response=MagicMock(status_code=429, headers={"retry-after": "60"}),
body={"error": {"message": "Rate limit exceeded", "type": "rate_limit_error"}}
)
with patch.object(client.chat.completions, "create", side_effect=rate_limit_error):
try:
client.chat.completions.create(
model="llama3-8b-8192",
messages=[{"role": "user", "content": "test"}]
)
pytest.fail("Expected RateLimitError")
except RateLimitError as e:
# Verify the error has useful information
assert e.response.status_code == 429
def test_context_length_error():
"""Oversized context should raise BadRequestError."""
client = Groq(api_key="test-key")
# Llama3-8b has 8192 token context window
context_error = BadRequestError(
message="Please reduce the length of the messages",
response=MagicMock(status_code=400),
body={"error": {"message": "Context length exceeded"}}
)
# Build a message that would exceed 8192 tokens
very_long_message = "word " * 10000
with patch.object(client.chat.completions, "create", side_effect=context_error):
with pytest.raises(BadRequestError) as exc_info:
client.chat.completions.create(
model="llama3-8b-8192",
messages=[{"role": "user", "content": very_long_message}]
)
assert "length" in str(exc_info.value).lower()
def test_network_error_retry_logic():
"""Connection errors should trigger retry logic."""
client = Groq(api_key="test-key")
connection_error = APIConnectionError(
message="Connection error",
request=MagicMock()
)
success_response = MagicMock()
success_response.choices = [MagicMock()]
success_response.choices[0].message.content = "Success after retry"
call_count = [0]
def flaky_call(**kwargs):
call_count[0] += 1
if call_count[0] < 3: # Fail first 2 attempts
raise connection_error
return success_response
with patch.object(client.chat.completions, "create", side_effect=flaky_call):
# Test your retry wrapper
def with_retry(fn, max_retries=3, **kwargs):
for attempt in range(max_retries):
try:
return fn(**kwargs)
except APIConnectionError:
if attempt == max_retries - 1:
raise
result = with_retry(
client.chat.completions.create,
model="llama3-8b-8192",
messages=[{"role": "user", "content": "test"}]
)
assert result.choices[0].message.content == "Success after retry"
assert call_count[0] == 3Testing JSON Mode
Groq supports JSON mode for structured output:
import json
def test_json_mode_returns_valid_json(groq_client):
"""JSON mode responses should be parseable JSON."""
mock_response = MagicMock()
mock_response.choices = [MagicMock()]
mock_response.choices[0].message.content = '{"name": "Alice", "age": 30}'
mock_response.choices[0].finish_reason = "stop"
with patch.object(groq_client.chat.completions, "create", return_value=mock_response):
response = groq_client.chat.completions.create(
model="llama3-8b-8192",
messages=[
{
"role": "system",
"content": "Respond with JSON only."
},
{
"role": "user",
"content": "Give me a person object with name and age."
}
],
response_format={"type": "json_object"}
)
content = response.choices[0].message.content
parsed = json.loads(content)
assert "name" in parsed
assert "age" in parsed
assert isinstance(parsed["age"], (int, float))
def test_json_mode_schema_validation(groq_client):
"""Validate JSON output matches expected schema."""
from jsonschema import validate, ValidationError
expected_schema = {
"type": "object",
"properties": {
"sentiment": {"type": "string", "enum": ["positive", "negative", "neutral"]},
"confidence": {"type": "number", "minimum": 0, "maximum": 1},
"reasoning": {"type": "string"}
},
"required": ["sentiment", "confidence", "reasoning"]
}
mock_json = {
"sentiment": "positive",
"confidence": 0.92,
"reasoning": "The text uses enthusiastic language."
}
mock_response = MagicMock()
mock_response.choices = [MagicMock()]
mock_response.choices[0].message.content = json.dumps(mock_json)
with patch.object(groq_client.chat.completions, "create", return_value=mock_response):
response = groq_client.chat.completions.create(
model="llama3-8b-8192",
messages=[{"role": "user", "content": "Analyze: I love this product!"}],
response_format={"type": "json_object"}
)
result = json.loads(response.choices[0].message.content)
validate(instance=result, schema=expected_schema) # Raises if invalidCI Configuration for Groq Tests
name: Groq Integration Tests
on:
push:
branches: [main, develop]
pull_request:
jobs:
unit-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install dependencies
run: pip install groq pytest pytest-asyncio pytest-cov jsonschema
- name: Run unit tests (mocked — no real API)
run: pytest tests/ -v -m "not benchmark and not integration"
env:
GROQ_API_KEY: "gsk_test_mock_key"
latency-benchmark:
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/main'
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Run latency benchmarks
run: pytest tests/ -v -m "benchmark" --benchmark-json=benchmark.json
env:
GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
- name: Upload benchmark results
uses: actions/upload-artifact@v4
with:
name: benchmark-results
path: benchmark.jsonContinuous Monitoring with HelpMeTest
Unit tests catch regressions before deploy. But Groq's performance can degrade due to factors outside your codebase — infrastructure changes, model updates, traffic congestion.
HelpMeTest lets you set up continuous end-to-end tests that verify your Groq integration is working correctly in production, 24/7. Write the test once in plain English: "Send a test completion request to our API and verify the response arrives within 2 seconds." HelpMeTest runs it every 5 minutes and alerts you the moment Groq latency spikes or errors appear.
Summary
Groq testing has three critical areas: correctness (does the response content match intent), streaming fidelity (are all chunks received in order), and latency (is the speed advantage maintained). With the pytest fixtures in this guide, you can achieve full coverage of all three without burning API quota in CI.
Key patterns to adopt:
- Mock at
client.chat.completions.create, not at the HTTP layer - Build
make_mock_stream()helpers for streaming tests - Track
x_groq.usagefields for latency assertions - Separate benchmark tests with a
benchmarkmarker to run only on main branch