Together AI Testing: Fine-Tuned Models, Batch Jobs, and Inference Testing

Together AI Testing: Fine-Tuned Models, Batch Jobs, and Inference Testing

Together AI positions itself as the platform for teams running custom fine-tuned models at scale. Beyond standard inference, it offers fine-tuning pipelines, batch inference for high-volume workloads, and a growing library of open-source models. Testing these features requires understanding Together AI's specific API patterns and failure modes.

This guide covers Together AI testing from basic inference to fine-tuned model deployment validation.

Together AI's Testing Challenges

Together AI introduces testing complexity that standard LLM providers don't have:

  • Fine-tuned model IDs: Custom models use long IDs like ft-xyz123-user-name/my-model:1234567890 — these need to be managed and validated
  • Batch jobs: Long-running async jobs need polling and timeout logic
  • Model availability: Open-source models can be cold or unavailable; tests must handle this
  • Rate limits by tier: Free and paid tiers have different limits that affect test design

Test Environment Setup

pip install together pytest pytest-asyncio httpx

conftest.py:

import pytest
from unittest.mock import MagicMock, patch
from together import Together, AsyncTogether

# Real Together AI model IDs for validation
TOGETHER_MODELS = [
    "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
    "mistralai/Mixtral-8x7B-Instruct-v0.1",
    "Qwen/Qwen2.5-72B-Instruct-Turbo",
    "meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo",
]

@pytest.fixture
def together_client():
    """Together AI client for testing."""
    return Together(api_key="test-api-key-not-real")

@pytest.fixture
def async_together_client():
    """Async Together AI client."""
    return AsyncTogether(api_key="test-api-key-not-real")

@pytest.fixture
def mock_completion():
    """Standard mock completion response."""
    response = MagicMock()
    response.id = "8781e98e-1234-5678-abcd-together001"
    response.model = "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo"
    response.choices = [MagicMock()]
    response.choices[0].index = 0
    response.choices[0].message.role = "assistant"
    response.choices[0].message.content = "Test response content"
    response.choices[0].finish_reason = "eos"
    response.usage = MagicMock()
    response.usage.prompt_tokens = 18
    response.usage.completion_tokens = 22
    response.usage.total_tokens = 40
    return response

@pytest.fixture
def fine_tuned_model_id():
    """Example fine-tuned model ID."""
    return "ft-abc123def456/user-org/my-custom-model:1700000000"

Testing Basic Inference

def test_basic_chat_completion(together_client, mock_completion):
    """Basic completion returns expected response structure."""
    with patch.object(together_client.chat.completions, "create",
                      return_value=mock_completion):
        response = together_client.chat.completions.create(
            model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
            messages=[{"role": "user", "content": "Hello"}]
        )
        
        assert response.choices[0].message.content == "Test response content"
        assert response.usage.total_tokens == 40

def test_llama_model_finish_reason(together_client):
    """Llama models use 'eos' finish reason, not 'stop'."""
    # Together AI uses 'eos' for most open-source models — different from OpenAI
    mock = MagicMock()
    mock.choices = [MagicMock()]
    mock.choices[0].finish_reason = "eos"  # Not "stop"!
    mock.choices[0].message.content = "Done"
    
    with patch.object(together_client.chat.completions, "create", return_value=mock):
        response = together_client.chat.completions.create(
            model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
            messages=[{"role": "user", "content": "test"}]
        )
        
        # Application code should handle both 'stop' and 'eos'
        finish_reason = response.choices[0].finish_reason
        assert finish_reason in ["stop", "eos", "length", "tool_calls"]

def test_temperature_and_top_p_parameters(together_client, mock_completion):
    """Sampling parameters should be forwarded correctly."""
    with patch.object(together_client.chat.completions, "create") as mock:
        mock.return_value = mock_completion
        
        together_client.chat.completions.create(
            model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
            messages=[{"role": "user", "content": "test"}],
            temperature=0.7,
            top_p=0.9,
            top_k=50,
            max_tokens=200
        )
        
        kwargs = mock.call_args.kwargs
        assert kwargs["temperature"] == 0.7
        assert kwargs["top_p"] == 0.9
        assert kwargs["top_k"] == 50
        assert kwargs["max_tokens"] == 200

def test_stop_sequences_passed(together_client, mock_completion):
    """Stop sequences should halt generation."""
    with patch.object(together_client.chat.completions, "create") as mock:
        mock.return_value = mock_completion
        
        together_client.chat.completions.create(
            model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
            messages=[{"role": "user", "content": "Count to 5: 1,"}],
            stop=["5", "\n\n"]
        )
        
        kwargs = mock.call_args.kwargs
        assert "stop" in kwargs
        assert "5" in kwargs["stop"]

Testing Fine-Tuned Models

Fine-tuned models need additional validation:

def test_fine_tuned_model_inference(together_client, fine_tuned_model_id):
    """Fine-tuned models should work with the same API as base models."""
    mock_response = MagicMock()
    mock_response.choices = [MagicMock()]
    mock_response.choices[0].message.content = "Fine-tuned response"
    mock_response.choices[0].message.role = "assistant"
    mock_response.choices[0].finish_reason = "eos"
    mock_response.model = fine_tuned_model_id
    
    with patch.object(together_client.chat.completions, "create",
                      return_value=mock_response):
        response = together_client.chat.completions.create(
            model=fine_tuned_model_id,
            messages=[{"role": "user", "content": "Classify: positive or negative?"}]
        )
        
        assert response.choices[0].message.content is not None
        assert response.model == fine_tuned_model_id

def test_fine_tuned_model_format_validation():
    """Fine-tuned model IDs should match Together's naming convention."""
    import re
    
    valid_patterns = [
        r"^ft-[a-z0-9]+/[a-z0-9-_]+/[a-z0-9-_]+:\d+$",  # Full path
        r"^[a-zA-Z0-9_-]+/[a-zA-Z0-9_.-]+$",              # org/model-name
    ]
    
    test_model_ids = {
        "ft-abc123/myorg/classifier:1700000000": True,
        "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo": True,
        "": False,
        "just-a-name-no-slash": False,
    }
    
    def is_valid_model_id(model_id: str) -> bool:
        if not model_id:
            return False
        return any(re.match(p, model_id) for p in valid_patterns)
    
    for model_id, expected in test_model_ids.items():
        result = is_valid_model_id(model_id)
        assert result == expected, f"Unexpected result for '{model_id}': got {result}"

def test_fine_tuning_dataset_format():
    """Validate training data format before submitting fine-tuning job."""
    import json
    
    # Together AI expects JSONL format with specific fields
    valid_sample = {
        "messages": [
            {"role": "system", "content": "You classify sentiment."},
            {"role": "user", "content": "I love this product!"},
            {"role": "assistant", "content": "positive"}
        ]
    }
    
    invalid_samples = [
        # Missing messages key
        {"text": "some text"},
        # Empty messages list
        {"messages": []},
        # Wrong role
        {"messages": [{"role": "human", "content": "hi"}]},
        # Missing content
        {"messages": [{"role": "user"}]},
    ]
    
    def validate_training_sample(sample: dict) -> tuple[bool, str]:
        if "messages" not in sample:
            return False, "Missing 'messages' key"
        if not sample["messages"]:
            return False, "Empty messages list"
        valid_roles = {"system", "user", "assistant"}
        for msg in sample["messages"]:
            if "role" not in msg or msg["role"] not in valid_roles:
                return False, f"Invalid role: {msg.get('role')}"
            if "content" not in msg:
                return False, "Missing content field"
        return True, "OK"
    
    valid, msg = validate_training_sample(valid_sample)
    assert valid, f"Valid sample failed: {msg}"
    
    for invalid in invalid_samples:
        valid, _ = validate_training_sample(invalid)
        assert not valid, f"Invalid sample should fail: {invalid}"

def test_fine_tuning_job_creation(together_client):
    """Fine-tuning job creation should return a job ID."""
    mock_job = MagicMock()
    mock_job.id = "ft-job-abc123"
    mock_job.status = "pending"
    mock_job.model = "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo"
    mock_job.created_at = 1700000000
    
    with patch.object(together_client.fine_tuning.jobs, "create",
                      return_value=mock_job) as mock:
        job = together_client.fine_tuning.jobs.create(
            training_file="file-abc123",
            model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
            n_epochs=3,
            batch_size=16,
            learning_rate=1e-5
        )
        
        assert job.id == "ft-job-abc123"
        assert job.status == "pending"
        
        call_kwargs = mock.call_args.kwargs
        assert call_kwargs["n_epochs"] == 3
        assert call_kwargs["batch_size"] == 16

Testing Batch Inference Jobs

Batch processing is Together AI's killer feature for high-volume workloads:

import time
from enum import Enum

class BatchJobStatus(Enum):
    QUEUED = "queued"
    RUNNING = "running"
    COMPLETED = "completed"
    FAILED = "failed"
    CANCELLED = "cancelled"

def test_batch_job_submission(together_client):
    """Batch job submission should return a job ID."""
    mock_batch_response = MagicMock()
    mock_batch_response.id = "batch-job-xyz789"
    mock_batch_response.status = BatchJobStatus.QUEUED.value
    mock_batch_response.created_at = 1700000000
    mock_batch_response.input_file_id = "file-input-abc"
    
    with patch.object(together_client.batches, "create",
                      return_value=mock_batch_response) as mock:
        batch_job = together_client.batches.create(
            input_file_id="file-input-abc",
            endpoint="/v1/chat/completions",
            completion_window="24h"
        )
        
        assert batch_job.id == "batch-job-xyz789"
        assert batch_job.status == "queued"

def test_batch_job_status_polling(together_client):
    """Status polling should handle queued → running → completed transitions."""
    statuses = ["queued", "running", "running", "completed"]
    call_count = [0]
    
    def mock_retrieve(job_id):
        mock = MagicMock()
        mock.id = job_id
        mock.status = statuses[min(call_count[0], len(statuses) - 1)]
        if mock.status == "completed":
            mock.output_file_id = "file-output-abc"
        call_count[0] += 1
        return mock
    
    with patch.object(together_client.batches, "retrieve", side_effect=mock_retrieve):
        def poll_until_complete(job_id, max_polls=10, interval_s=0):
            for _ in range(max_polls):
                job = together_client.batches.retrieve(job_id)
                if job.status in ("completed", "failed", "cancelled"):
                    return job
                time.sleep(interval_s)
            raise TimeoutError("Batch job did not complete in time")
        
        result = poll_until_complete("batch-job-xyz789")
        
        assert result.status == "completed"
        assert result.output_file_id == "file-output-abc"
        assert call_count[0] == 4  # Polled 4 times

def test_batch_job_failure_handling(together_client):
    """Failed batch jobs should raise appropriate exceptions."""
    mock_failed_job = MagicMock()
    mock_failed_job.id = "batch-job-failed"
    mock_failed_job.status = "failed"
    mock_failed_job.error_file_id = "file-errors-abc"
    mock_failed_job.errors = [{"message": "Invalid input format", "line": 42}]
    
    with patch.object(together_client.batches, "retrieve",
                      return_value=mock_failed_job):
        job = together_client.batches.retrieve("batch-job-failed")
        
        assert job.status == "failed"
        assert job.error_file_id is not None
        
        # Application should handle failed jobs
        if job.status == "failed":
            errors = job.errors or []
            error_messages = [e.get("message", "") for e in errors]
            assert len(error_messages) > 0

def test_batch_results_parsing(together_client):
    """Batch output file should be parseable JSONL."""
    import json
    from io import StringIO
    
    # Together AI batch output is JSONL
    batch_output_lines = [
        '{"id": "req-001", "custom_id": "item-1", "response": {"body": {"choices": [{"message": {"content": "Response 1"}}]}}}',
        '{"id": "req-002", "custom_id": "item-2", "response": {"body": {"choices": [{"message": {"content": "Response 2"}}]}}}',
        '{"id": "req-003", "custom_id": "item-3", "error": {"message": "Context too long"}}'
    ]
    
    results = []
    errors = []
    
    for line in batch_output_lines:
        item = json.loads(line)
        if "error" in item:
            errors.append(item)
        else:
            content = item["response"]["body"]["choices"][0]["message"]["content"]
            results.append({"id": item["custom_id"], "content": content})
    
    assert len(results) == 2
    assert len(errors) == 1
    assert results[0]["content"] == "Response 1"
    assert errors[0]["custom_id"] == "item-3"

Testing Streaming with Together AI

def make_together_stream(tokens: list[str]):
    """Create mock Together AI streaming response."""
    for i, token in enumerate(tokens):
        chunk = MagicMock()
        chunk.id = "stream-test"
        chunk.choices = [MagicMock()]
        chunk.choices[0].delta.content = token
        chunk.choices[0].delta.role = "assistant" if i == 0 else None
        chunk.choices[0].finish_reason = "eos" if i == len(tokens) - 1 else None
        yield chunk

def test_streaming_collects_tokens(together_client):
    """All streaming tokens should be collected in order."""
    tokens = ["The", " sky", " is", " blue", "."]
    
    with patch.object(together_client.chat.completions, "create",
                      return_value=make_together_stream(tokens)):
        stream = together_client.chat.completions.create(
            model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
            messages=[{"role": "user", "content": "What color is the sky?"}],
            stream=True
        )
        
        full_response = ""
        for chunk in stream:
            if chunk.choices[0].delta.content:
                full_response += chunk.choices[0].delta.content
        
        assert full_response == "The sky is blue."

def test_streaming_with_repetition_penalty(together_client):
    """Repetition penalty parameter should be forwarded."""
    with patch.object(together_client.chat.completions, "create") as mock:
        mock.return_value = make_together_stream(["test"])
        
        together_client.chat.completions.create(
            model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
            messages=[{"role": "user", "content": "test"}],
            repetition_penalty=1.1,
            stream=True
        )
        
        kwargs = mock.call_args.kwargs
        assert kwargs.get("repetition_penalty") == 1.1

Testing Image Generation (FLUX)

Together AI also hosts image generation models:

def test_image_generation_request(together_client):
    """Image generation should return image data."""
    mock_image_response = MagicMock()
    mock_image_response.data = [MagicMock()]
    mock_image_response.data[0].url = "https://together-storage.com/image-abc.png"
    mock_image_response.data[0].b64_json = None
    
    with patch.object(together_client.images, "generate",
                      return_value=mock_image_response) as mock:
        response = together_client.images.generate(
            prompt="A sunset over mountains",
            model="black-forest-labs/FLUX.1-schnell-Free",
            width=1024,
            height=1024,
            steps=4,
            n=1
        )
        
        assert len(response.data) == 1
        assert response.data[0].url.startswith("https://")
        
        kwargs = mock.call_args.kwargs
        assert kwargs["width"] == 1024
        assert kwargs["steps"] == 4

def test_image_generation_b64_format(together_client):
    """Base64 format returns b64_json instead of URL."""
    mock_response = MagicMock()
    mock_response.data = [MagicMock()]
    mock_response.data[0].url = None
    mock_response.data[0].b64_json = "iVBORw0KGgoAAAANSUhEUgAA..."
    
    with patch.object(together_client.images, "generate",
                      return_value=mock_response):
        response = together_client.images.generate(
            prompt="A cat",
            model="black-forest-labs/FLUX.1-schnell-Free",
            response_format="b64_json"
        )
        
        assert response.data[0].b64_json is not None
        assert response.data[0].url is None

Error Handling for Together AI

from together.error import (
    AuthenticationError,
    RateLimitError,
    InvalidRequestError,
    APIError
)

def test_model_not_found_error(together_client):
    """Requesting nonexistent model raises InvalidRequestError."""
    with patch.object(together_client.chat.completions, "create",
                      side_effect=InvalidRequestError(
                          message="Model not found: nonexistent/model",
                          status_code=404
                      )):
        with pytest.raises(InvalidRequestError) as exc_info:
            together_client.chat.completions.create(
                model="nonexistent/model",
                messages=[{"role": "user", "content": "test"}]
            )
        assert "not found" in str(exc_info.value).lower()

def test_rate_limit_backoff(together_client, mock_completion):
    """Rate limit errors should trigger exponential backoff."""
    call_count = [0]
    
    def flaky_call(**kwargs):
        call_count[0] += 1
        if call_count[0] <= 2:
            raise RateLimitError(
                message="Rate limit exceeded",
                status_code=429
            )
        return mock_completion
    
    with patch.object(together_client.chat.completions, "create",
                      side_effect=flaky_call):
        def with_backoff(fn, max_retries=3, **kwargs):
            for attempt in range(max_retries):
                try:
                    return fn(**kwargs)
                except RateLimitError:
                    if attempt < max_retries - 1:
                        time.sleep(0)  # Would be 2**attempt in production
                    else:
                        raise
        
        result = with_backoff(
            together_client.chat.completions.create,
            model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
            messages=[{"role": "user", "content": "test"}]
        )
        
        assert result.choices[0].message.content == "Test response content"
        assert call_count[0] == 3

CI/CD Configuration

name: Together AI Tests

on: [push, pull_request]

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      
      - name: Install
        run: pip install together pytest pytest-asyncio pytest-cov
      
      - name: Run tests
        run: pytest tests/ -v --cov=src
        env:
          TOGETHER_API_KEY: "test-key-mock"
  
  integration-tests:
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      
      - name: Install
        run: pip install together pytest pytest-asyncio
      
      - name: Run integration tests
        run: pytest tests/ -m integration -v --timeout=120
        env:
          TOGETHER_API_KEY: ${{ secrets.TOGETHER_API_KEY }}

Monitoring Together AI in Production

Together AI's open-source model ecosystem means model availability can shift. Models get updated, deprecated, or moved to different serving infrastructure. A fine-tuned model that worked perfectly yesterday can become unavailable after a platform update.

HelpMeTest provides continuous end-to-end monitoring that catches these issues automatically. Configure a test that hits your Together AI integration every 5 minutes, validates the response format matches your expectations, and fires an alert if your fine-tuned model returns unexpected results. A 14-day free trial makes it easy to confirm the 24/7 monitoring catches most availability issues before users notice them.

Summary

Together AI testing has three distinct layers that most providers don't have:

  1. Standard inference — completions, streaming, sampling parameters (note eos vs stop)
  2. Fine-tuned model validation — ID format, dataset format, job submission and polling
  3. Batch processing — job lifecycle, JSONL output parsing, error handling per-item

The most common production failure mode is fine-tuned model ID drift — a model is retrained with a new timestamp suffix and the old ID returns a 404. Build validation tests that verify your model ID resolves correctly on every deployment.

Read more

Start now free