Testing

Testing with ChromaDB: Collections, Embeddings, and Persistence

HelpMeTest

HelpMeTest

— 6 min read
Testing with ChromaDB: Collections, Embeddings, and Persistence

ChromaDB is one of the most popular open-source vector databases for AI applications — lightweight, easy to embed in Python applications, and well-suited for testing because it runs fully in-memory without external dependencies.

This guide covers testing ChromaDB integrations: collection management, embedding functions, metadata filtering, and persistence.

Why ChromaDB Is Testing-Friendly

ChromaDB can run in three modes:

  • In-memory: No persistence, perfect for unit tests
  • Persistent: Writes to disk, for integration tests
  • HTTP client: Connects to a running Chroma server, for E2E tests

This makes it easy to test each layer of your application independently. Use in-memory mode for fast unit tests, persistent mode for integration tests, and the HTTP client for production smoke tests.

Installation

pip install chromadb pytest pytest-asyncio

Testing Collection Management

# tests/test_chroma_collections.py
import pytest
import chromadb
from chromadb.config import Settings

@pytest.fixture
def chroma_client():
    """Fresh in-memory Chroma client for each test."""
    client = chromadb.Client()  # In-memory, ephemeral
    yield client
    # No cleanup needed — in-memory client resets automatically

class TestCollectionManagement:
    def test_create_collection(self, chroma_client):
        """Creating a collection should succeed and be retrievable."""
        collection = chroma_client.create_collection("test-docs")
        
        assert collection.name == "test-docs"
        assert collection.count() == 0

    def test_get_or_create_idempotent(self, chroma_client):
        """get_or_create_collection should not raise when collection exists."""
        first = chroma_client.get_or_create_collection("my-collection")
        second = chroma_client.get_or_create_collection("my-collection")
        
        assert first.name == second.name

    def test_list_collections(self, chroma_client):
        """Created collections should appear in list_collections."""
        chroma_client.create_collection("collection-a")
        chroma_client.create_collection("collection-b")
        
        collections = chroma_client.list_collections()
        names = [c.name for c in collections]
        
        assert "collection-a" in names
        assert "collection-b" in names

    def test_delete_collection(self, chroma_client):
        """Deleted collection should not appear in list."""
        chroma_client.create_collection("temp-collection")
        chroma_client.delete_collection("temp-collection")
        
        collections = chroma_client.list_collections()
        names = [c.name for c in collections]
        
        assert "temp-collection" not in names

    def test_delete_nonexistent_collection_raises(self, chroma_client):
        """Deleting a non-existent collection should raise ValueError."""
        with pytest.raises(Exception):  # ValueError or similar
            chroma_client.delete_collection("does-not-exist")

    def test_collection_count_updates_on_add(self, chroma_client):
        """Document count should increase after adding documents."""
        collection = chroma_client.create_collection("count-test")
        
        collection.add(
            ids=["doc-1", "doc-2", "doc-3"],
            documents=["Text one", "Text two", "Text three"],
        )
        
        assert collection.count() == 3

Testing Embedding Functions

ChromaDB supports custom embedding functions. Test that they produce correct outputs:

# tests/test_embedding_functions.py
import pytest
import chromadb
import numpy as np
from chromadb.utils import embedding_functions

class TestDefaultEmbeddingFunction:
    @pytest.fixture
    def collection_with_default_ef(self):
        """Collection using ChromaDB's default all-MiniLM-L6-v2 embedding."""
        client = chromadb.Client()
        ef = embedding_functions.DefaultEmbeddingFunction()
        return client.create_collection("default-ef-test", embedding_function=ef)

    def test_adds_documents_and_generates_embeddings(self, collection_with_default_ef):
        """Documents should be embedded and stored without specifying embeddings manually."""
        collection_with_default_ef.add(
            ids=["doc-1"],
            documents=["Python testing with pytest and fixtures"],
        )
        
        assert collection_with_default_ef.count() == 1

    def test_query_by_text_uses_embedding_function(self, collection_with_default_ef):
        """Querying by text should use the embedding function automatically."""
        collection_with_default_ef.add(
            ids=["python-testing", "ml-deploy", "cooking"],
            documents=[
                "Python unit testing and pytest fixtures",
                "Machine learning model deployment",
                "How to cook pasta carbonara",
            ],
        )
        
        results = collection_with_default_ef.query(
            query_texts=["automated testing in Python"],
            n_results=2,
        )
        
        top_ids = results["ids"][0]
        assert "python-testing" in top_ids, (
            f"Expected python-testing in top-2, got {top_ids}"
        )

class TestCustomEmbeddingFunction:
    def test_custom_embedding_function_is_called(self):
        """Custom EF should be invoked when documents are added."""
        call_count = 0
        
        class CountingEF(chromadb.EmbeddingFunction):
            def __call__(self, input):
                nonlocal call_count
                call_count += 1
                # Return fixed 4-dim embeddings for simplicity
                return [[0.1, 0.2, 0.3, 0.4] for _ in input]
        
        client = chromadb.Client()
        collection = client.create_collection("custom-ef-test", embedding_function=CountingEF())
        
        collection.add(ids=["doc-1", "doc-2"], documents=["Text A", "Text B"])
        
        assert call_count > 0

    def test_custom_embedding_dimensions(self):
        """Collection should correctly store embeddings with custom dimensions."""
        class FixedDimensionEF(chromadb.EmbeddingFunction):
            def __call__(self, input):
                return [[float(i) for i in range(8)] for _ in input]  # 8-dim
        
        client = chromadb.Client()
        collection = client.create_collection("8dim-test", embedding_function=FixedDimensionEF())
        
        collection.add(ids=["doc-1"], documents=["Test document"])
        
        # Retrieve and verify embedding dimensions
        result = collection.get(ids=["doc-1"], include=["embeddings"])
        embedding = result["embeddings"][0]
        
        assert len(embedding) == 8

Testing Metadata Filtering

ChromaDB's where filters let you combine semantic search with structured filtering:

# tests/test_metadata_filtering.py
import pytest
import chromadb

@pytest.fixture
def filtered_collection():
    """Collection with varied metadata for filter testing."""
    client = chromadb.Client()
    collection = client.create_collection("filter-test")
    
    collection.add(
        ids=["doc-1", "doc-2", "doc-3", "doc-4", "doc-5"],
        documents=[
            "Python testing best practices",
            "JavaScript end-to-end testing",
            "Python performance optimization",
            "JavaScript React component testing",
            "Database testing strategies",
        ],
        metadatas=[
            {"language": "python", "category": "testing", "year": 2024},
            {"language": "javascript", "category": "testing", "year": 2024},
            {"language": "python", "category": "performance", "year": 2023},
            {"language": "javascript", "category": "testing", "year": 2025},
            {"language": "sql", "category": "testing", "year": 2024},
        ],
    )
    
    return collection

class TestMetadataFiltering:
    def test_filter_by_exact_value(self, filtered_collection):
        """$eq filter should return only matching documents."""
        results = filtered_collection.get(
            where={"language": {"$eq": "python"}}
        )
        
        assert len(results["ids"]) == 2
        assert set(results["ids"]) == {"doc-1", "doc-3"}

    def test_filter_by_multiple_conditions(self, filtered_collection):
        """$and filter should return docs matching all conditions."""
        results = filtered_collection.get(
            where={
                "$and": [
                    {"language": {"$eq": "javascript"}},
                    {"category": {"$eq": "testing"}},
                ]
            }
        )
        
        assert len(results["ids"]) == 2
        assert set(results["ids"]) == {"doc-2", "doc-4"}

    def test_filter_by_year_range(self, filtered_collection):
        """Numeric comparison filter should work correctly."""
        results = filtered_collection.get(
            where={"year": {"$gte": 2024}}
        )
        
        # 2024 and 2025 docs: doc-1, doc-2, doc-4, doc-5
        assert len(results["ids"]) == 4

    def test_filter_with_query(self, filtered_collection):
        """Combining filter with semantic query should narrow results."""
        results = filtered_collection.query(
            query_texts=["testing"],
            n_results=3,
            where={"language": {"$eq": "python"}},
        )
        
        returned_ids = results["ids"][0]
        
        # All returned docs should be python docs
        assert all(
            doc_id in {"doc-1", "doc-3"} for doc_id in returned_ids
        )

    def test_where_document_text_filter(self, filtered_collection):
        """where_document filter should match on document text content."""
        results = filtered_collection.get(
            where_document={"$contains": "JavaScript"}
        )
        
        assert len(results["ids"]) == 2
        assert set(results["ids"]) == {"doc-2", "doc-4"}

    def test_not_in_filter(self, filtered_collection):
        """$nin filter should exclude specified values."""
        results = filtered_collection.get(
            where={"language": {"$nin": ["python", "javascript"]}}
        )
        
        # Only SQL doc remains
        assert len(results["ids"]) == 1
        assert results["ids"][0] == "doc-5"

Testing Persistence

# tests/test_chroma_persistence.py
import pytest
import tempfile
import os
import chromadb

@pytest.fixture
def persistent_db_path():
    """Create a temporary directory for persistent storage."""
    with tempfile.TemporaryDirectory() as tmpdir:
        yield tmpdir

class TestPersistence:
    def test_data_persists_across_client_instances(self, persistent_db_path):
        """Data written to a persistent client should be readable by a new client instance."""
        # Write data
        client_1 = chromadb.PersistentClient(path=persistent_db_path)
        collection = client_1.create_collection("persistent-test")
        collection.add(
            ids=["doc-1", "doc-2"],
            documents=["First document", "Second document"],
            metadatas=[{"index": 1}, {"index": 2}],
        )
        del client_1  # Close the first client
        
        # Read data with a new client instance
        client_2 = chromadb.PersistentClient(path=persistent_db_path)
        loaded_collection = client_2.get_collection("persistent-test")
        
        assert loaded_collection.count() == 2
        
        result = loaded_collection.get(ids=["doc-1"])
        assert result["documents"][0] == "First document"

    def test_updates_persist(self, persistent_db_path):
        """Updates to existing documents should persist across sessions."""
        client_1 = chromadb.PersistentClient(path=persistent_db_path)
        collection = client_1.create_collection("update-test")
        
        # Initial add
        collection.add(ids=["doc-1"], documents=["Original text"])
        
        # Update
        collection.update(
            ids=["doc-1"],
            documents=["Updated text"],
            metadatas=[{"updated": True}],
        )
        del client_1
        
        client_2 = chromadb.PersistentClient(path=persistent_db_path)
        loaded = client_2.get_collection("update-test")
        result = loaded.get(ids=["doc-1"])
        
        assert result["documents"][0] == "Updated text"
        assert result["metadatas"][0]["updated"] is True

    def test_deletes_persist(self, persistent_db_path):
        """Deleted documents should not appear after reloading."""
        client_1 = chromadb.PersistentClient(path=persistent_db_path)
        collection = client_1.create_collection("delete-test")
        
        collection.add(
            ids=["keep-1", "delete-1"],
            documents=["Keep this", "Delete this"],
        )
        collection.delete(ids=["delete-1"])
        del client_1
        
        client_2 = chromadb.PersistentClient(path=persistent_db_path)
        loaded = client_2.get_collection("delete-test")
        
        assert loaded.count() == 1
        result = loaded.get()
        assert "delete-1" not in result["ids"]

Testing CRUD Operations

# tests/test_chroma_crud.py
import pytest
import chromadb

@pytest.fixture
def collection():
    client = chromadb.Client()
    coll = client.create_collection("crud-test")
    yield coll

class TestCRUDOperations:
    def test_add_and_get_by_id(self, collection):
        collection.add(
            ids=["item-1"],
            documents=["Test content"],
            metadatas=[{"tag": "test"}],
        )
        
        result = collection.get(ids=["item-1"])
        
        assert result["ids"] == ["item-1"]
        assert result["documents"] == ["Test content"]
        assert result["metadatas"][0]["tag"] == "test"

    def test_upsert_creates_if_not_exists(self, collection):
        collection.upsert(
            ids=["new-doc"],
            documents=["New document content"],
        )
        
        assert collection.count() == 1

    def test_upsert_updates_if_exists(self, collection):
        collection.add(ids=["existing"], documents=["Original"])
        collection.upsert(ids=["existing"], documents=["Updated"])
        
        result = collection.get(ids=["existing"])
        assert result["documents"][0] == "Updated"
        assert collection.count() == 1  # Count unchanged

    def test_update_metadata_without_changing_document(self, collection):
        collection.add(
            ids=["doc-1"],
            documents=["Document text"],
            metadatas=[{"version": 1}],
        )
        
        collection.update(ids=["doc-1"], metadatas=[{"version": 2}])
        
        result = collection.get(ids=["doc-1"])
        assert result["metadatas"][0]["version"] == 2
        assert result["documents"][0] == "Document text"  # Unchanged

    def test_delete_specific_ids(self, collection):
        collection.add(
            ids=["keep", "delete"],
            documents=["Keep this", "Delete this"],
        )
        
        collection.delete(ids=["delete"])
        
        assert collection.count() == 1
        result = collection.get()
        assert "delete" not in result["ids"]

    def test_get_with_include_embeddings(self, collection):
        collection.add(ids=["doc-1"], documents=["Test text"])
        
        result = collection.get(ids=["doc-1"], include=["embeddings", "documents"])
        
        assert result["embeddings"] is not None
        assert len(result["embeddings"][0]) > 0  # Non-empty embedding vector

Integration with HelpMeTest

ChromaDB tests validate that your vector store works correctly in isolation. But production RAG applications fail in integrated ways — the embedding model changes, metadata filtering has edge cases, or persistence fails under load. HelpMeTest monitors your ChromaDB-backed application continuously, running automated tests against the live system and alerting you when retrieval quality drops.

Summary

Testing ChromaDB integrations covers:

  • Collection management: Create, list, delete, count
  • Embedding functions: Custom EFs are called, correct dimensions stored
  • Metadata filtering: $eq, $gte, $nin, $and, $or, where_document
  • Persistence: Data survives client recreation — adds, updates, and deletes
  • CRUD: Add, get, upsert, update, delete — all behave correctly

ChromaDB's in-memory mode makes these tests extremely fast — a full suite covering collection management, filtering, and CRUD typically runs in under 5 seconds. Use them as pre-commit checks to catch integration issues before they reach production.

Read more