Groq API Testing: Streaming, Latency Benchmarks, and Error Handling

Groq API Testing: Streaming, Latency Benchmarks, and Error Handling

Groq stands apart in the LLM landscape for one reason: speed. The Language Processing Unit (LPU) architecture delivers token generation that's often 10-20x faster than GPU-based inference. But raw speed creates new testing challenges — your test suite needs to validate not just correctness but latency characteristics, streaming behavior, and the specific error modes that come with an infrastructure built for throughput.

This guide covers everything you need to test Groq API integrations in production-grade Python applications.

Why Groq Testing Is Unique

Testing Groq differs from testing other LLM providers because:

  • Latency is a feature: Users choose Groq for speed. A correct but slow response is a regression.
  • Streaming is the default path: Most Groq use cases use streaming; testing non-streaming is not enough.
  • Context limits are strict: Groq's models have specific context windows. Tests need to exercise boundary conditions.
  • Rate limits hit differently: At 30 req/min for free tier, rate limit errors happen in tests if you're not careful.

Setting Up Groq Test Environment

pip install groq pytest pytest-asyncio httpx

conftest.py:

import pytest
from unittest.mock import MagicMock, patch
from groq import Groq, AsyncGroq

@pytest.fixture
def groq_client():
    """Groq client configured for testing."""
    return Groq(api_key="gsk_test_key_never_real")

@pytest.fixture
def async_groq_client():
    """Async Groq client for async tests."""
    return AsyncGroq(api_key="gsk_test_key_never_real")

@pytest.fixture
def mock_groq_response():
    """Standard mock Groq completion response."""
    response = MagicMock()
    response.id = "chatcmpl-test-123"
    response.model = "llama3-8b-8192"
    response.choices = [MagicMock()]
    response.choices[0].index = 0
    response.choices[0].message.role = "assistant"
    response.choices[0].message.content = "Test response content"
    response.choices[0].finish_reason = "stop"
    response.usage.prompt_tokens = 15
    response.usage.completion_tokens = 25
    response.usage.total_tokens = 40
    response.x_groq = MagicMock()
    response.x_groq.usage.queue_time = 0.003
    response.x_groq.usage.prompt_time = 0.012
    response.x_groq.usage.completion_time = 0.025
    response.x_groq.usage.total_time = 0.040
    return response

Testing Groq Completion Calls

import pytest
from unittest.mock import patch, MagicMock
from groq import Groq

def test_basic_chat_completion(groq_client, mock_groq_response):
    """Basic completion call returns expected structure."""
    with patch.object(groq_client.chat.completions, "create", return_value=mock_groq_response):
        response = groq_client.chat.completions.create(
            model="llama3-8b-8192",
            messages=[{"role": "user", "content": "Hello"}]
        )
        
        assert response.choices[0].message.content == "Test response content"
        assert response.choices[0].message.role == "assistant"
        assert response.usage.total_tokens == 40

def test_model_selection():
    """Verify correct model is passed to the API."""
    client = Groq(api_key="test-key")
    target_model = "mixtral-8x7b-32768"
    
    with patch.object(client.chat.completions, "create") as mock_create:
        mock_create.return_value = MagicMock(
            choices=[MagicMock(message=MagicMock(content="ok", role="assistant"))]
        )
        
        client.chat.completions.create(
            model=target_model,
            messages=[{"role": "user", "content": "test"}]
        )
        
        call_kwargs = mock_create.call_args.kwargs
        assert call_kwargs["model"] == target_model

def test_system_prompt_included():
    """System prompts should be passed as the first message."""
    client = Groq(api_key="test-key")
    
    with patch.object(client.chat.completions, "create") as mock_create:
        mock_create.return_value = MagicMock(
            choices=[MagicMock(message=MagicMock(content="response", role="assistant"))]
        )
        
        client.chat.completions.create(
            model="llama3-8b-8192",
            messages=[
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": "What time is it?"}
            ]
        )
        
        messages = mock_create.call_args.kwargs["messages"]
        assert messages[0]["role"] == "system"
        assert "helpful assistant" in messages[0]["content"]

def test_temperature_and_max_tokens_passed():
    """Generation parameters should be forwarded correctly."""
    client = Groq(api_key="test-key")
    
    with patch.object(client.chat.completions, "create") as mock_create:
        mock_create.return_value = MagicMock(
            choices=[MagicMock(message=MagicMock(content="ok", role="assistant"))]
        )
        
        client.chat.completions.create(
            model="llama3-8b-8192",
            messages=[{"role": "user", "content": "test"}],
            temperature=0.2,
            max_tokens=500
        )
        
        kwargs = mock_create.call_args.kwargs
        assert kwargs["temperature"] == 0.2
        assert kwargs["max_tokens"] == 500

Testing Groq Streaming Responses

Streaming is where Groq shines — and where tests get complex:

import time
from typing import Iterator

def make_mock_stream(chunks: list[str]) -> Iterator:
    """Create a mock streaming response."""
    for i, text in enumerate(chunks):
        chunk = MagicMock()
        chunk.id = "chatcmpl-test"
        chunk.model = "llama3-8b-8192"
        chunk.choices = [MagicMock()]
        chunk.choices[0].index = 0
        chunk.choices[0].delta.role = "assistant" if i == 0 else None
        chunk.choices[0].delta.content = text
        chunk.choices[0].finish_reason = None if i < len(chunks) - 1 else "stop"
        yield chunk

def test_streaming_collects_all_chunks(groq_client):
    """All stream chunks should be received and assembled."""
    expected_tokens = ["Hello", ", ", "world", "!"]
    
    with patch.object(groq_client.chat.completions, "create",
                      return_value=make_mock_stream(expected_tokens)):
        stream = groq_client.chat.completions.create(
            model="llama3-8b-8192",
            messages=[{"role": "user", "content": "Say hello"}],
            stream=True
        )
        
        collected = ""
        for chunk in stream:
            if chunk.choices[0].delta.content:
                collected += chunk.choices[0].delta.content
        
        assert collected == "Hello, world!"

def test_streaming_finish_reason():
    """Stream should end with finish_reason='stop' on normal completion."""
    client = Groq(api_key="test-key")
    chunks = ["Final", " chunk"]
    
    with patch.object(client.chat.completions, "create",
                      return_value=make_mock_stream(chunks)):
        stream = client.chat.completions.create(
            model="llama3-8b-8192",
            messages=[{"role": "user", "content": "test"}],
            stream=True
        )
        
        last_chunk = None
        for chunk in stream:
            last_chunk = chunk
        
        assert last_chunk is not None
        assert last_chunk.choices[0].finish_reason == "stop"

@pytest.mark.asyncio
async def test_async_streaming(async_groq_client):
    """Async streaming should work with async for loops."""
    async def async_stream():
        for chunk in make_mock_stream(["async ", "response"]):
            yield chunk
    
    with patch.object(async_groq_client.chat.completions, "create",
                      return_value=async_stream()):
        collected = ""
        async with async_groq_client.chat.completions.create(
            model="llama3-8b-8192",
            messages=[{"role": "user", "content": "test"}],
            stream=True
        ) as stream:
            async for chunk in stream:
                if chunk.choices[0].delta.content:
                    collected += chunk.choices[0].delta.content
        
        assert collected == "async response"

def test_streaming_with_first_token_latency():
    """Measure time-to-first-token for streaming responses."""
    client = Groq(api_key="test-key")
    
    def slow_stream():
        time.sleep(0.01)  # Simulate network latency
        yield from make_mock_stream(["First", " token"])
    
    with patch.object(client.chat.completions, "create", return_value=slow_stream()):
        start = time.monotonic()
        stream = client.chat.completions.create(
            model="llama3-8b-8192",
            messages=[{"role": "user", "content": "test"}],
            stream=True
        )
        
        first_token_time = None
        for chunk in stream:
            if chunk.choices[0].delta.content and first_token_time is None:
                first_token_time = time.monotonic() - start
        
        assert first_token_time is not None
        # In production, you'd assert first_token_time < your_sla_threshold

Latency Benchmarking

Groq's selling point is speed. Here's how to benchmark it properly:

import statistics
import time
import pytest

class LatencyBenchmark:
    """Measure and assert latency characteristics."""
    
    def __init__(self):
        self.measurements = []
    
    def record(self, duration_ms: float):
        self.measurements.append(duration_ms)
    
    @property
    def p50(self) -> float:
        return statistics.median(self.measurements)
    
    @property
    def p95(self) -> float:
        sorted_m = sorted(self.measurements)
        idx = int(len(sorted_m) * 0.95)
        return sorted_m[min(idx, len(sorted_m) - 1)]
    
    @property
    def p99(self) -> float:
        sorted_m = sorted(self.measurements)
        idx = int(len(sorted_m) * 0.99)
        return sorted_m[min(idx, len(sorted_m) - 1)]

def test_latency_benchmark_structure():
    """Verify benchmark captures statistics correctly."""
    bench = LatencyBenchmark()
    
    # Simulate 10 measurements
    for ms in [45, 52, 48, 61, 43, 49, 55, 47, 50, 200]:  # 200ms outlier
        bench.record(ms)
    
    assert bench.p50 < 60  # Median well under 60ms
    assert bench.p95 < 150  # 95th percentile under 150ms
    assert bench.p99 <= 200  # 99th under 200ms (our outlier)

@pytest.mark.benchmark
def test_groq_tokens_per_second(groq_client):
    """Calculate tokens per second from Groq response metadata."""
    mock_response = MagicMock()
    mock_response.choices = [MagicMock()]
    mock_response.choices[0].message.content = "Test response " * 50
    mock_response.usage.completion_tokens = 100
    mock_response.x_groq = MagicMock()
    mock_response.x_groq.usage.completion_time = 0.5  # 0.5 seconds
    
    with patch.object(groq_client.chat.completions, "create", return_value=mock_response):
        response = groq_client.chat.completions.create(
            model="llama3-8b-8192",
            messages=[{"role": "user", "content": "Write a short paragraph."}]
        )
        
        completion_tokens = response.usage.completion_tokens
        completion_time = response.x_groq.usage.completion_time
        tps = completion_tokens / completion_time
        
        # Groq typically delivers 200-800 TPS depending on model
        # This asserts the calculation is reasonable
        assert tps > 0
        assert tps < 2000  # Sanity upper bound

Testing Error Handling

Groq-specific errors need explicit test coverage:

from groq import (
    AuthenticationError,
    RateLimitError,
    APIStatusError,
    APIConnectionError,
    BadRequestError
)
import groq

def test_authentication_error_raised():
    """Invalid API key should raise AuthenticationError."""
    client = Groq(api_key="invalid-key")
    
    with patch.object(client.chat.completions, "create",
                      side_effect=AuthenticationError(
                          message="Invalid API Key",
                          response=MagicMock(status_code=401),
                          body={"error": {"message": "Invalid API Key"}}
                      )):
        with pytest.raises(AuthenticationError) as exc_info:
            client.chat.completions.create(
                model="llama3-8b-8192",
                messages=[{"role": "user", "content": "test"}]
            )
        assert "Invalid API Key" in str(exc_info.value)

def test_rate_limit_error_handling():
    """Rate limit errors should be caught and handled gracefully."""
    client = Groq(api_key="test-key")
    
    rate_limit_error = RateLimitError(
        message="Rate limit exceeded for requests per minute",
        response=MagicMock(status_code=429, headers={"retry-after": "60"}),
        body={"error": {"message": "Rate limit exceeded", "type": "rate_limit_error"}}
    )
    
    with patch.object(client.chat.completions, "create", side_effect=rate_limit_error):
        try:
            client.chat.completions.create(
                model="llama3-8b-8192",
                messages=[{"role": "user", "content": "test"}]
            )
            pytest.fail("Expected RateLimitError")
        except RateLimitError as e:
            # Verify the error has useful information
            assert e.response.status_code == 429

def test_context_length_error():
    """Oversized context should raise BadRequestError."""
    client = Groq(api_key="test-key")
    
    # Llama3-8b has 8192 token context window
    context_error = BadRequestError(
        message="Please reduce the length of the messages",
        response=MagicMock(status_code=400),
        body={"error": {"message": "Context length exceeded"}}
    )
    
    # Build a message that would exceed 8192 tokens
    very_long_message = "word " * 10000
    
    with patch.object(client.chat.completions, "create", side_effect=context_error):
        with pytest.raises(BadRequestError) as exc_info:
            client.chat.completions.create(
                model="llama3-8b-8192",
                messages=[{"role": "user", "content": very_long_message}]
            )
        assert "length" in str(exc_info.value).lower()

def test_network_error_retry_logic():
    """Connection errors should trigger retry logic."""
    client = Groq(api_key="test-key")
    
    connection_error = APIConnectionError(
        message="Connection error",
        request=MagicMock()
    )
    
    success_response = MagicMock()
    success_response.choices = [MagicMock()]
    success_response.choices[0].message.content = "Success after retry"
    
    call_count = [0]
    
    def flaky_call(**kwargs):
        call_count[0] += 1
        if call_count[0] < 3:  # Fail first 2 attempts
            raise connection_error
        return success_response
    
    with patch.object(client.chat.completions, "create", side_effect=flaky_call):
        # Test your retry wrapper
        def with_retry(fn, max_retries=3, **kwargs):
            for attempt in range(max_retries):
                try:
                    return fn(**kwargs)
                except APIConnectionError:
                    if attempt == max_retries - 1:
                        raise
            
        result = with_retry(
            client.chat.completions.create,
            model="llama3-8b-8192",
            messages=[{"role": "user", "content": "test"}]
        )
        
        assert result.choices[0].message.content == "Success after retry"
        assert call_count[0] == 3

Testing JSON Mode

Groq supports JSON mode for structured output:

import json

def test_json_mode_returns_valid_json(groq_client):
    """JSON mode responses should be parseable JSON."""
    mock_response = MagicMock()
    mock_response.choices = [MagicMock()]
    mock_response.choices[0].message.content = '{"name": "Alice", "age": 30}'
    mock_response.choices[0].finish_reason = "stop"
    
    with patch.object(groq_client.chat.completions, "create", return_value=mock_response):
        response = groq_client.chat.completions.create(
            model="llama3-8b-8192",
            messages=[
                {
                    "role": "system",
                    "content": "Respond with JSON only."
                },
                {
                    "role": "user",
                    "content": "Give me a person object with name and age."
                }
            ],
            response_format={"type": "json_object"}
        )
        
        content = response.choices[0].message.content
        parsed = json.loads(content)
        
        assert "name" in parsed
        assert "age" in parsed
        assert isinstance(parsed["age"], (int, float))

def test_json_mode_schema_validation(groq_client):
    """Validate JSON output matches expected schema."""
    from jsonschema import validate, ValidationError
    
    expected_schema = {
        "type": "object",
        "properties": {
            "sentiment": {"type": "string", "enum": ["positive", "negative", "neutral"]},
            "confidence": {"type": "number", "minimum": 0, "maximum": 1},
            "reasoning": {"type": "string"}
        },
        "required": ["sentiment", "confidence", "reasoning"]
    }
    
    mock_json = {
        "sentiment": "positive",
        "confidence": 0.92,
        "reasoning": "The text uses enthusiastic language."
    }
    
    mock_response = MagicMock()
    mock_response.choices = [MagicMock()]
    mock_response.choices[0].message.content = json.dumps(mock_json)
    
    with patch.object(groq_client.chat.completions, "create", return_value=mock_response):
        response = groq_client.chat.completions.create(
            model="llama3-8b-8192",
            messages=[{"role": "user", "content": "Analyze: I love this product!"}],
            response_format={"type": "json_object"}
        )
        
        result = json.loads(response.choices[0].message.content)
        validate(instance=result, schema=expected_schema)  # Raises if invalid

CI Configuration for Groq Tests

name: Groq Integration Tests

on:
  push:
    branches: [main, develop]
  pull_request:

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      
      - name: Install dependencies
        run: pip install groq pytest pytest-asyncio pytest-cov jsonschema
      
      - name: Run unit tests (mocked — no real API)
        run: pytest tests/ -v -m "not benchmark and not integration"
        env:
          GROQ_API_KEY: "gsk_test_mock_key"

  latency-benchmark:
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      
      - name: Run latency benchmarks
        run: pytest tests/ -v -m "benchmark" --benchmark-json=benchmark.json
        env:
          GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
      
      - name: Upload benchmark results
        uses: actions/upload-artifact@v4
        with:
          name: benchmark-results
          path: benchmark.json

Continuous Monitoring with HelpMeTest

Unit tests catch regressions before deploy. But Groq's performance can degrade due to factors outside your codebase — infrastructure changes, model updates, traffic congestion.

HelpMeTest lets you set up continuous end-to-end tests that verify your Groq integration is working correctly in production, 24/7. Write the test once in plain English: "Send a test completion request to our API and verify the response arrives within 2 seconds." HelpMeTest runs it every 5 minutes and alerts you the moment Groq latency spikes or errors appear.

Summary

Groq testing has three critical areas: correctness (does the response content match intent), streaming fidelity (are all chunks received in order), and latency (is the speed advantage maintained). With the pytest fixtures in this guide, you can achieve full coverage of all three without burning API quota in CI.

Key patterns to adopt:

  • Mock at client.chat.completions.create, not at the HTTP layer
  • Build make_mock_stream() helpers for streaming tests
  • Track x_groq.usage fields for latency assertions
  • Separate benchmark tests with a benchmark marker to run only on main branch

Read more

Start now free