Together AI Testing: Fine-Tuned Models, Batch Jobs, and Inference Testing
Together AI positions itself as the platform for teams running custom fine-tuned models at scale. Beyond standard inference, it offers fine-tuning pipelines, batch inference for high-volume workloads, and a growing library of open-source models. Testing these features requires understanding Together AI's specific API patterns and failure modes.
This guide covers Together AI testing from basic inference to fine-tuned model deployment validation.
Together AI's Testing Challenges
Together AI introduces testing complexity that standard LLM providers don't have:
- Fine-tuned model IDs: Custom models use long IDs like
ft-xyz123-user-name/my-model:1234567890— these need to be managed and validated - Batch jobs: Long-running async jobs need polling and timeout logic
- Model availability: Open-source models can be cold or unavailable; tests must handle this
- Rate limits by tier: Free and paid tiers have different limits that affect test design
Test Environment Setup
pip install together pytest pytest-asyncio httpxconftest.py:
import pytest
from unittest.mock import MagicMock, patch
from together import Together, AsyncTogether
# Real Together AI model IDs for validation
TOGETHER_MODELS = [
"meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
"mistralai/Mixtral-8x7B-Instruct-v0.1",
"Qwen/Qwen2.5-72B-Instruct-Turbo",
"meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo",
]
@pytest.fixture
def together_client():
"""Together AI client for testing."""
return Together(api_key="test-api-key-not-real")
@pytest.fixture
def async_together_client():
"""Async Together AI client."""
return AsyncTogether(api_key="test-api-key-not-real")
@pytest.fixture
def mock_completion():
"""Standard mock completion response."""
response = MagicMock()
response.id = "8781e98e-1234-5678-abcd-together001"
response.model = "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo"
response.choices = [MagicMock()]
response.choices[0].index = 0
response.choices[0].message.role = "assistant"
response.choices[0].message.content = "Test response content"
response.choices[0].finish_reason = "eos"
response.usage = MagicMock()
response.usage.prompt_tokens = 18
response.usage.completion_tokens = 22
response.usage.total_tokens = 40
return response
@pytest.fixture
def fine_tuned_model_id():
"""Example fine-tuned model ID."""
return "ft-abc123def456/user-org/my-custom-model:1700000000"Testing Basic Inference
def test_basic_chat_completion(together_client, mock_completion):
"""Basic completion returns expected response structure."""
with patch.object(together_client.chat.completions, "create",
return_value=mock_completion):
response = together_client.chat.completions.create(
model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
messages=[{"role": "user", "content": "Hello"}]
)
assert response.choices[0].message.content == "Test response content"
assert response.usage.total_tokens == 40
def test_llama_model_finish_reason(together_client):
"""Llama models use 'eos' finish reason, not 'stop'."""
# Together AI uses 'eos' for most open-source models — different from OpenAI
mock = MagicMock()
mock.choices = [MagicMock()]
mock.choices[0].finish_reason = "eos" # Not "stop"!
mock.choices[0].message.content = "Done"
with patch.object(together_client.chat.completions, "create", return_value=mock):
response = together_client.chat.completions.create(
model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
messages=[{"role": "user", "content": "test"}]
)
# Application code should handle both 'stop' and 'eos'
finish_reason = response.choices[0].finish_reason
assert finish_reason in ["stop", "eos", "length", "tool_calls"]
def test_temperature_and_top_p_parameters(together_client, mock_completion):
"""Sampling parameters should be forwarded correctly."""
with patch.object(together_client.chat.completions, "create") as mock:
mock.return_value = mock_completion
together_client.chat.completions.create(
model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
messages=[{"role": "user", "content": "test"}],
temperature=0.7,
top_p=0.9,
top_k=50,
max_tokens=200
)
kwargs = mock.call_args.kwargs
assert kwargs["temperature"] == 0.7
assert kwargs["top_p"] == 0.9
assert kwargs["top_k"] == 50
assert kwargs["max_tokens"] == 200
def test_stop_sequences_passed(together_client, mock_completion):
"""Stop sequences should halt generation."""
with patch.object(together_client.chat.completions, "create") as mock:
mock.return_value = mock_completion
together_client.chat.completions.create(
model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
messages=[{"role": "user", "content": "Count to 5: 1,"}],
stop=["5", "\n\n"]
)
kwargs = mock.call_args.kwargs
assert "stop" in kwargs
assert "5" in kwargs["stop"]Testing Fine-Tuned Models
Fine-tuned models need additional validation:
def test_fine_tuned_model_inference(together_client, fine_tuned_model_id):
"""Fine-tuned models should work with the same API as base models."""
mock_response = MagicMock()
mock_response.choices = [MagicMock()]
mock_response.choices[0].message.content = "Fine-tuned response"
mock_response.choices[0].message.role = "assistant"
mock_response.choices[0].finish_reason = "eos"
mock_response.model = fine_tuned_model_id
with patch.object(together_client.chat.completions, "create",
return_value=mock_response):
response = together_client.chat.completions.create(
model=fine_tuned_model_id,
messages=[{"role": "user", "content": "Classify: positive or negative?"}]
)
assert response.choices[0].message.content is not None
assert response.model == fine_tuned_model_id
def test_fine_tuned_model_format_validation():
"""Fine-tuned model IDs should match Together's naming convention."""
import re
valid_patterns = [
r"^ft-[a-z0-9]+/[a-z0-9-_]+/[a-z0-9-_]+:\d+$", # Full path
r"^[a-zA-Z0-9_-]+/[a-zA-Z0-9_.-]+$", # org/model-name
]
test_model_ids = {
"ft-abc123/myorg/classifier:1700000000": True,
"meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo": True,
"": False,
"just-a-name-no-slash": False,
}
def is_valid_model_id(model_id: str) -> bool:
if not model_id:
return False
return any(re.match(p, model_id) for p in valid_patterns)
for model_id, expected in test_model_ids.items():
result = is_valid_model_id(model_id)
assert result == expected, f"Unexpected result for '{model_id}': got {result}"
def test_fine_tuning_dataset_format():
"""Validate training data format before submitting fine-tuning job."""
import json
# Together AI expects JSONL format with specific fields
valid_sample = {
"messages": [
{"role": "system", "content": "You classify sentiment."},
{"role": "user", "content": "I love this product!"},
{"role": "assistant", "content": "positive"}
]
}
invalid_samples = [
# Missing messages key
{"text": "some text"},
# Empty messages list
{"messages": []},
# Wrong role
{"messages": [{"role": "human", "content": "hi"}]},
# Missing content
{"messages": [{"role": "user"}]},
]
def validate_training_sample(sample: dict) -> tuple[bool, str]:
if "messages" not in sample:
return False, "Missing 'messages' key"
if not sample["messages"]:
return False, "Empty messages list"
valid_roles = {"system", "user", "assistant"}
for msg in sample["messages"]:
if "role" not in msg or msg["role"] not in valid_roles:
return False, f"Invalid role: {msg.get('role')}"
if "content" not in msg:
return False, "Missing content field"
return True, "OK"
valid, msg = validate_training_sample(valid_sample)
assert valid, f"Valid sample failed: {msg}"
for invalid in invalid_samples:
valid, _ = validate_training_sample(invalid)
assert not valid, f"Invalid sample should fail: {invalid}"
def test_fine_tuning_job_creation(together_client):
"""Fine-tuning job creation should return a job ID."""
mock_job = MagicMock()
mock_job.id = "ft-job-abc123"
mock_job.status = "pending"
mock_job.model = "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo"
mock_job.created_at = 1700000000
with patch.object(together_client.fine_tuning.jobs, "create",
return_value=mock_job) as mock:
job = together_client.fine_tuning.jobs.create(
training_file="file-abc123",
model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
n_epochs=3,
batch_size=16,
learning_rate=1e-5
)
assert job.id == "ft-job-abc123"
assert job.status == "pending"
call_kwargs = mock.call_args.kwargs
assert call_kwargs["n_epochs"] == 3
assert call_kwargs["batch_size"] == 16Testing Batch Inference Jobs
Batch processing is Together AI's killer feature for high-volume workloads:
import time
from enum import Enum
class BatchJobStatus(Enum):
QUEUED = "queued"
RUNNING = "running"
COMPLETED = "completed"
FAILED = "failed"
CANCELLED = "cancelled"
def test_batch_job_submission(together_client):
"""Batch job submission should return a job ID."""
mock_batch_response = MagicMock()
mock_batch_response.id = "batch-job-xyz789"
mock_batch_response.status = BatchJobStatus.QUEUED.value
mock_batch_response.created_at = 1700000000
mock_batch_response.input_file_id = "file-input-abc"
with patch.object(together_client.batches, "create",
return_value=mock_batch_response) as mock:
batch_job = together_client.batches.create(
input_file_id="file-input-abc",
endpoint="/v1/chat/completions",
completion_window="24h"
)
assert batch_job.id == "batch-job-xyz789"
assert batch_job.status == "queued"
def test_batch_job_status_polling(together_client):
"""Status polling should handle queued → running → completed transitions."""
statuses = ["queued", "running", "running", "completed"]
call_count = [0]
def mock_retrieve(job_id):
mock = MagicMock()
mock.id = job_id
mock.status = statuses[min(call_count[0], len(statuses) - 1)]
if mock.status == "completed":
mock.output_file_id = "file-output-abc"
call_count[0] += 1
return mock
with patch.object(together_client.batches, "retrieve", side_effect=mock_retrieve):
def poll_until_complete(job_id, max_polls=10, interval_s=0):
for _ in range(max_polls):
job = together_client.batches.retrieve(job_id)
if job.status in ("completed", "failed", "cancelled"):
return job
time.sleep(interval_s)
raise TimeoutError("Batch job did not complete in time")
result = poll_until_complete("batch-job-xyz789")
assert result.status == "completed"
assert result.output_file_id == "file-output-abc"
assert call_count[0] == 4 # Polled 4 times
def test_batch_job_failure_handling(together_client):
"""Failed batch jobs should raise appropriate exceptions."""
mock_failed_job = MagicMock()
mock_failed_job.id = "batch-job-failed"
mock_failed_job.status = "failed"
mock_failed_job.error_file_id = "file-errors-abc"
mock_failed_job.errors = [{"message": "Invalid input format", "line": 42}]
with patch.object(together_client.batches, "retrieve",
return_value=mock_failed_job):
job = together_client.batches.retrieve("batch-job-failed")
assert job.status == "failed"
assert job.error_file_id is not None
# Application should handle failed jobs
if job.status == "failed":
errors = job.errors or []
error_messages = [e.get("message", "") for e in errors]
assert len(error_messages) > 0
def test_batch_results_parsing(together_client):
"""Batch output file should be parseable JSONL."""
import json
from io import StringIO
# Together AI batch output is JSONL
batch_output_lines = [
'{"id": "req-001", "custom_id": "item-1", "response": {"body": {"choices": [{"message": {"content": "Response 1"}}]}}}',
'{"id": "req-002", "custom_id": "item-2", "response": {"body": {"choices": [{"message": {"content": "Response 2"}}]}}}',
'{"id": "req-003", "custom_id": "item-3", "error": {"message": "Context too long"}}'
]
results = []
errors = []
for line in batch_output_lines:
item = json.loads(line)
if "error" in item:
errors.append(item)
else:
content = item["response"]["body"]["choices"][0]["message"]["content"]
results.append({"id": item["custom_id"], "content": content})
assert len(results) == 2
assert len(errors) == 1
assert results[0]["content"] == "Response 1"
assert errors[0]["custom_id"] == "item-3"Testing Streaming with Together AI
def make_together_stream(tokens: list[str]):
"""Create mock Together AI streaming response."""
for i, token in enumerate(tokens):
chunk = MagicMock()
chunk.id = "stream-test"
chunk.choices = [MagicMock()]
chunk.choices[0].delta.content = token
chunk.choices[0].delta.role = "assistant" if i == 0 else None
chunk.choices[0].finish_reason = "eos" if i == len(tokens) - 1 else None
yield chunk
def test_streaming_collects_tokens(together_client):
"""All streaming tokens should be collected in order."""
tokens = ["The", " sky", " is", " blue", "."]
with patch.object(together_client.chat.completions, "create",
return_value=make_together_stream(tokens)):
stream = together_client.chat.completions.create(
model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
messages=[{"role": "user", "content": "What color is the sky?"}],
stream=True
)
full_response = ""
for chunk in stream:
if chunk.choices[0].delta.content:
full_response += chunk.choices[0].delta.content
assert full_response == "The sky is blue."
def test_streaming_with_repetition_penalty(together_client):
"""Repetition penalty parameter should be forwarded."""
with patch.object(together_client.chat.completions, "create") as mock:
mock.return_value = make_together_stream(["test"])
together_client.chat.completions.create(
model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
messages=[{"role": "user", "content": "test"}],
repetition_penalty=1.1,
stream=True
)
kwargs = mock.call_args.kwargs
assert kwargs.get("repetition_penalty") == 1.1Testing Image Generation (FLUX)
Together AI also hosts image generation models:
def test_image_generation_request(together_client):
"""Image generation should return image data."""
mock_image_response = MagicMock()
mock_image_response.data = [MagicMock()]
mock_image_response.data[0].url = "https://together-storage.com/image-abc.png"
mock_image_response.data[0].b64_json = None
with patch.object(together_client.images, "generate",
return_value=mock_image_response) as mock:
response = together_client.images.generate(
prompt="A sunset over mountains",
model="black-forest-labs/FLUX.1-schnell-Free",
width=1024,
height=1024,
steps=4,
n=1
)
assert len(response.data) == 1
assert response.data[0].url.startswith("https://")
kwargs = mock.call_args.kwargs
assert kwargs["width"] == 1024
assert kwargs["steps"] == 4
def test_image_generation_b64_format(together_client):
"""Base64 format returns b64_json instead of URL."""
mock_response = MagicMock()
mock_response.data = [MagicMock()]
mock_response.data[0].url = None
mock_response.data[0].b64_json = "iVBORw0KGgoAAAANSUhEUgAA..."
with patch.object(together_client.images, "generate",
return_value=mock_response):
response = together_client.images.generate(
prompt="A cat",
model="black-forest-labs/FLUX.1-schnell-Free",
response_format="b64_json"
)
assert response.data[0].b64_json is not None
assert response.data[0].url is NoneError Handling for Together AI
from together.error import (
AuthenticationError,
RateLimitError,
InvalidRequestError,
APIError
)
def test_model_not_found_error(together_client):
"""Requesting nonexistent model raises InvalidRequestError."""
with patch.object(together_client.chat.completions, "create",
side_effect=InvalidRequestError(
message="Model not found: nonexistent/model",
status_code=404
)):
with pytest.raises(InvalidRequestError) as exc_info:
together_client.chat.completions.create(
model="nonexistent/model",
messages=[{"role": "user", "content": "test"}]
)
assert "not found" in str(exc_info.value).lower()
def test_rate_limit_backoff(together_client, mock_completion):
"""Rate limit errors should trigger exponential backoff."""
call_count = [0]
def flaky_call(**kwargs):
call_count[0] += 1
if call_count[0] <= 2:
raise RateLimitError(
message="Rate limit exceeded",
status_code=429
)
return mock_completion
with patch.object(together_client.chat.completions, "create",
side_effect=flaky_call):
def with_backoff(fn, max_retries=3, **kwargs):
for attempt in range(max_retries):
try:
return fn(**kwargs)
except RateLimitError:
if attempt < max_retries - 1:
time.sleep(0) # Would be 2**attempt in production
else:
raise
result = with_backoff(
together_client.chat.completions.create,
model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
messages=[{"role": "user", "content": "test"}]
)
assert result.choices[0].message.content == "Test response content"
assert call_count[0] == 3CI/CD Configuration
name: Together AI Tests
on: [push, pull_request]
jobs:
unit-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install
run: pip install together pytest pytest-asyncio pytest-cov
- name: Run tests
run: pytest tests/ -v --cov=src
env:
TOGETHER_API_KEY: "test-key-mock"
integration-tests:
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install
run: pip install together pytest pytest-asyncio
- name: Run integration tests
run: pytest tests/ -m integration -v --timeout=120
env:
TOGETHER_API_KEY: ${{ secrets.TOGETHER_API_KEY }}Monitoring Together AI in Production
Together AI's open-source model ecosystem means model availability can shift. Models get updated, deprecated, or moved to different serving infrastructure. A fine-tuned model that worked perfectly yesterday can become unavailable after a platform update.
HelpMeTest provides continuous end-to-end monitoring that catches these issues automatically. Configure a test that hits your Together AI integration every 5 minutes, validates the response format matches your expectations, and fires an alert if your fine-tuned model returns unexpected results. A 14-day free trial makes it easy to confirm the 24/7 monitoring catches most availability issues before users notice them.
Summary
Together AI testing has three distinct layers that most providers don't have:
- Standard inference — completions, streaming, sampling parameters (note
eosvsstop) - Fine-tuned model validation — ID format, dataset format, job submission and polling
- Batch processing — job lifecycle, JSONL output parsing, error handling per-item
The most common production failure mode is fine-tuned model ID drift — a model is retrained with a new timestamp suffix and the old ID returns a 404. Build validation tests that verify your model ID resolves correctly on every deployment.