Testing

Vector Database Testing: Embedding Quality, Similarity, and Recall

HelpMeTest

HelpMeTest

— 8 min read
Vector Database Testing: Embedding Quality, Similarity, and Recall

Vector databases are the retrieval engine at the heart of every RAG system. And yet they are among the least tested components in most AI stacks. Teams validate their LLM outputs extensively but trust the vector store to just work. This trust is misplaced.

Embedding models drift when you upgrade them. Similarity thresholds that work for English queries fail for multilingual input. Indexing performance degrades under load. Retrieval recall can drop by 15 points after a routine knowledge base update, with no alerts, no logs, and no obvious indication anything changed.

This guide covers how to systematically test vector databases — whether you are using Pinecone, Weaviate, Chroma, or pgvector — and how to build continuous monitoring that catches these problems before users do.

What You Are Actually Testing

Vector database testing has four distinct concern areas, each requiring different test approaches:

  1. Embedding quality: Does the model that converts text to vectors produce semantically meaningful representations? Do similar texts get similar embeddings?
  2. Similarity calibration: Are your similarity thresholds set correctly? A threshold of 0.7 might be right for one embedding model and completely wrong for another.
  3. Retrieval recall@k: When you query, does the relevant content actually appear in the top k results?
  4. Indexing correctness: Is what you put in actually retrievable? Are updates reflected correctly? Are deletions propagated?

Let us work through each.

Testing Embedding Quality

Embedding quality is foundational. Bad embeddings make every downstream metric worse and no amount of RAG tuning can compensate.

Semantic Similarity Tests

Verify that your embedding model captures meaning correctly for your domain. Generic models trained on web text may perform poorly on specialized technical or medical content:

from sentence_transformers import SentenceTransformer, util
import pytest

model = SentenceTransformer("text-embedding-3-small")  # Or your model

SEMANTIC_PAIRS = [
    # Should be similar
    {
        "text_a": "How do I reset my password?",
        "text_b": "I forgot my password and need to create a new one",
        "expected_similarity": "high",
        "min_score": 0.75
    },
    # Should be dissimilar
    {
        "text_a": "How do I reset my password?",
        "text_b": "What is the capital of France?",
        "expected_similarity": "low",
        "max_score": 0.5
    },
    # Domain-specific: verify your model handles your vocabulary
    {
        "text_a": "pgvector HNSW index performance",
        "text_b": "approximate nearest neighbor search speed in Postgres",
        "expected_similarity": "high",
        "min_score": 0.70
    }
]

@pytest.mark.parametrize("pair", SEMANTIC_PAIRS)
def test_embedding_semantic_quality(pair):
    emb_a = model.encode(pair["text_a"], convert_to_tensor=True)
    emb_b = model.encode(pair["text_b"], convert_to_tensor=True)
    score = util.cos_sim(emb_a, emb_b).item()
    
    if pair["expected_similarity"] == "high":
        assert score >= pair["min_score"], (
            f"Texts should be similar but got {score:.3f}:\n"
            f"  A: {pair['text_a']}\n  B: {pair['text_b']}"
        )
    else:
        assert score <= pair["max_score"], (
            f"Texts should be dissimilar but got {score:.3f}:\n"
            f"  A: {pair['text_a']}\n  B: {pair['text_b']}"
        )

Embedding Dimension and Format Validation

Simple but often skipped — validate that embeddings have the expected shape and value range:

def test_embedding_format(embedder):
    text = "Sample document for testing"
    embedding = embedder.embed(text)
    
    assert len(embedding) == 1536, f"Expected 1536 dimensions, got {len(embedding)}"
    assert all(isinstance(v, float) for v in embedding), "All values should be floats"
    assert -1.0 <= min(embedding) <= max(embedding) <= 1.0, (
        "Embedding values should be in [-1, 1] range for normalized embeddings"
    )
    
    # Check for NaN or Inf (common when input is empty or malformed)
    import math
    assert not any(math.isnan(v) or math.isinf(v) for v in embedding), (
        "Embedding contains NaN or Inf values"
    )

Empty and Edge Case Inputs

Embedding models can return garbage for edge cases that then pollute your vector store:

@pytest.mark.parametrize("text,should_raise", [
    ("", True),
    (" " * 100, True),   # Whitespace only
    ("a", False),        # Single character
    ("." * 5000, False), # Very long repetitive text
    ("SELECT * FROM users; DROP TABLE users;", False),  # SQL injection attempt
    ("🚀🎯💡🔥", False),  # Emoji only
])
def test_embedding_edge_cases(text, should_raise, embedder):
    if should_raise:
        with pytest.raises((ValueError, Exception)):
            embedder.embed(text)
    else:
        embedding = embedder.embed(text)
        assert len(embedding) > 0
        import math
        assert not any(math.isnan(v) for v in embedding)

Validating Similarity Thresholds

Every vector database query involves a similarity threshold or a top-k cutoff. The right threshold depends on your embedding model, your document corpus, and your query patterns. There is no universal correct value.

Threshold Calibration Test

Find the threshold that maximizes the F1 score on your golden dataset:

import numpy as np
from sklearn.metrics import f1_score

def calibrate_similarity_threshold(
    vector_store,
    golden_pairs: list[dict],  # [{query, doc_id, is_relevant}]
    k: int = 10,
    threshold_range: tuple = (0.3, 0.9),
    steps: int = 60
) -> dict:
    thresholds = np.linspace(threshold_range[0], threshold_range[1], steps)
    results = []
    
    for threshold in thresholds:
        y_true = []
        y_pred = []
        
        for pair in golden_pairs:
            hits = vector_store.search(
                query=pair["query"],
                k=k,
                score_threshold=threshold
            )
            hit_ids = {h.id for h in hits}
            
            y_true.append(int(pair["is_relevant"]))
            y_pred.append(int(pair["doc_id"] in hit_ids))
        
        f1 = f1_score(y_true, y_pred, zero_division=0)
        results.append({"threshold": threshold, "f1": f1})
    
    best = max(results, key=lambda r: r["f1"])
    return {
        "optimal_threshold": best["threshold"],
        "optimal_f1": best["f1"],
        "all_results": results
    }

Run this calibration whenever you change your embedding model or significantly update your corpus. The optimal threshold can shift substantially between model versions.

Threshold Regression Test

Once calibrated, lock the threshold and alert on regression:

CALIBRATED_THRESHOLD = 0.72  # From your calibration run
CALIBRATED_F1 = 0.88

def test_similarity_threshold_stable(vector_store, golden_pairs):
    results = calibrate_similarity_threshold(
        vector_store, golden_pairs, 
        threshold_range=(CALIBRATED_THRESHOLD - 0.1, CALIBRATED_THRESHOLD + 0.1),
        steps=20
    )
    
    assert results["optimal_f1"] >= CALIBRATED_F1 - 0.05, (
        f"Similarity threshold F1 regressed: {results['optimal_f1']:.3f} "
        f"vs baseline {CALIBRATED_F1:.3f}"
    )

Measuring Retrieval Recall@k

Recall@k is the fraction of relevant documents that appear in the top k results. It is the single most important metric for a vector database.

Per-Vector-Store Recall Benchmark

Here is a framework for testing recall across different vector stores. Run this when evaluating which database to use, and again in CI to catch regressions:

# Abstraction layer for testing multiple stores
class VectorStoreAdapter:
    def upsert(self, id: str, embedding: list[float], metadata: dict): ...
    def search(self, embedding: list[float], k: int) -> list: ...
    def delete(self, id: str): ...

class PineconeAdapter(VectorStoreAdapter):
    def __init__(self, index):
        self.index = index
    
    def upsert(self, id, embedding, metadata):
        self.index.upsert(vectors=[(id, embedding, metadata)])
    
    def search(self, embedding, k):
        result = self.index.query(vector=embedding, top_k=k, include_metadata=True)
        return [{"id": m.id, "score": m.score} for m in result.matches]

class ChromaAdapter(VectorStoreAdapter):
    def __init__(self, collection):
        self.collection = collection
    
    def upsert(self, id, embedding, metadata):
        self.collection.upsert(ids=[id], embeddings=[embedding], metadatas=[metadata])
    
    def search(self, embedding, k):
        result = self.collection.query(query_embeddings=[embedding], n_results=k)
        ids = result["ids"][0]
        distances = result["distances"][0]
        return [{"id": i, "score": 1 - d} for i, d in zip(ids, distances)]

def measure_recall_at_k(adapter: VectorStoreAdapter, golden_dataset: list[dict], k: int) -> float:
    hits = 0
    total = 0
    
    for case in golden_dataset:
        query_emb = embed(case["query"])
        results = adapter.search(query_emb, k=k)
        result_ids = {r["id"] for r in results}
        
        relevant_ids = set(case["relevant_doc_ids"])
        hits += len(result_ids & relevant_ids)
        total += len(relevant_ids)
    
    return hits / total if total > 0 else 0.0

@pytest.mark.parametrize("k", [1, 3, 5, 10])
def test_recall_at_k(vector_store_adapter, golden_dataset, k):
    recall = measure_recall_at_k(vector_store_adapter, golden_dataset, k)
    min_recall = {1: 0.50, 3: 0.70, 5: 0.80, 10: 0.90}[k]
    
    assert recall >= min_recall, (
        f"Recall@{k} = {recall:.3f}, below minimum {min_recall}"
    )

pgvector-Specific Testing

If you are using pgvector with PostgreSQL, test the index configuration specifically. The choice between IVFFlat and HNSW, and the lists or m/ef_construction parameters, directly affect recall:

def test_pgvector_hnsw_recall(pg_connection, golden_queries):
    """HNSW should give near-exact recall at the cost of build time."""
    cursor = pg_connection.cursor()
    
    cursor.execute("""
        CREATE INDEX IF NOT EXISTS test_hnsw_idx 
        ON documents USING hnsw (embedding vector_cosine_ops)
        WITH (m = 16, ef_construction = 64)
    """)
    pg_connection.commit()
    cursor.execute("SET hnsw.ef_search = 100")
    
    recall = measure_recall_via_sql(cursor, golden_queries, k=5)
    assert recall >= 0.95, f"HNSW recall@5 too low: {recall:.3f}"

def test_pgvector_ivfflat_recall(pg_connection, golden_queries):
    """IVFFlat is faster but trades off some recall — verify it is acceptable."""
    cursor = pg_connection.cursor()
    
    cursor.execute("""
        CREATE INDEX IF NOT EXISTS test_ivf_idx
        ON documents USING ivfflat (embedding vector_cosine_ops)
        WITH (lists = 100)
    """)
    pg_connection.commit()
    cursor.execute("SET ivfflat.probes = 10")
    
    recall = measure_recall_via_sql(cursor, golden_queries, k=5)
    assert recall >= 0.85, f"IVFFlat recall@5 too low: {recall:.3f}. Increase probes or lists."

Testing Indexing Performance

Retrieval quality degrades under certain indexing patterns. Test these explicitly:

Bulk Insert Correctness

After a bulk insert, verify all documents are retrievable:

def test_bulk_insert_completeness(vector_store, test_documents, embedder):
    """All inserted documents should be retrievable."""
    for doc in test_documents:
        emb = embedder.embed(doc["text"])
        vector_store.upsert(id=doc["id"], embedding=emb, metadata={"text": doc["text"]})
    
    missing = []
    for doc in test_documents:
        emb = embedder.embed(doc["text"])
        results = vector_store.search(emb, k=1)
        
        if not results or results[0]["id"] != doc["id"]:
            missing.append(doc["id"])
    
    assert len(missing) == 0, f"Documents not retrievable after insert: {missing}"

Update and Delete Propagation

Updates and deletes must be reflected immediately. This is a common failure mode in vector databases with eventual consistency:

def test_update_propagation(vector_store, embedder):
    original_text = "The price is $10 per month"
    updated_text = "The price is $25 per month"
    doc_id = "pricing-doc-001"
    
    vector_store.upsert(
        id=doc_id,
        embedding=embedder.embed(original_text),
        metadata={"text": original_text}
    )
    vector_store.upsert(
        id=doc_id,
        embedding=embedder.embed(updated_text),
        metadata={"text": updated_text}
    )
    
    results = vector_store.search(embedder.embed("What is the current pricing?"), k=5)
    matching = [r for r in results if r["id"] == doc_id]
    
    assert len(matching) == 1, "Document should appear exactly once"
    assert matching[0]["metadata"]["text"] == updated_text

def test_delete_propagation(vector_store, embedder):
    text = "Confidential: employee salary data"
    doc_id = "confidential-hr-001"
    
    vector_store.upsert(id=doc_id, embedding=embedder.embed(text), metadata={})
    vector_store.delete(id=doc_id)
    
    results = vector_store.search(embedder.embed(text), k=10)
    deleted_ids = [r["id"] for r in results if r["id"] == doc_id]
    
    assert len(deleted_ids) == 0, (
        "Deleted document still appears in search results. "
        "This could expose sensitive content."
    )

Detecting Embedding Drift

Embedding drift occurs when the model used to create stored vectors differs from the model used to embed queries. This can happen after a model upgrade, an API change, or even a silent model update from a hosted provider.

Drift Detection Test

def test_no_embedding_dimension_drift(vector_store, current_embedder):
    """
    Sample stored vectors and compare dimensions to current model output.
    Mismatch means vectors were created with a different model.
    """
    sample_vectors = vector_store.sample(n=10)
    stored_dims = set(len(v["embedding"]) for v in sample_vectors)
    
    current_dim = len(current_embedder.embed("test"))
    
    assert stored_dims == {current_dim}, (
        f"Dimension mismatch — stored vectors have dims {stored_dims}, "
        f"current model produces {current_dim}. "
        f"Re-embed the corpus before querying."
    )

Distribution Shift Detection

Embedding distributions can drift even with the same model version if the corpus changes significantly:

import numpy as np
from scipy import stats

def test_embedding_distribution_stable(corpus_sample, embedder, baseline_norms):
    """
    Check that embedding norm distribution has not shifted significantly.
    Uses KS test to compare current distribution to baseline.
    """
    current_embeddings = [embedder.embed(doc) for doc in corpus_sample]
    current_norms = [np.linalg.norm(e) for e in current_embeddings]
    
    ks_stat, p_value = stats.ks_2samp(baseline_norms, current_norms)
    
    assert p_value > 0.05, (
        f"Embedding distribution has shifted significantly (KS p={p_value:.4f}). "
        f"Consider re-evaluating similarity thresholds."
    )

CI Integration

Structure your vector database tests in three tiers for CI efficiency:

# .github/workflows/vector-db-tests.yml
name: Vector DB Tests

jobs:
  # Tier 1: Every commit (< 30 seconds)
  unit:
    steps:
      - run: pytest tests/vector_db/ -m unit
  
  # Tier 2: Every PR (2-5 minutes)
  integration:
    steps:
      - run: pytest tests/vector_db/ -m integration
  
  # Tier 3: Nightly (10-30 minutes)
  full-evaluation:
    if: github.event_name == 'schedule'
    steps:
      - run: pytest tests/vector_db/ --tb=long --json-report

Tier 1 (every commit): Embedding format validation, edge case inputs, delete propagation on small fixtures.

Tier 2 (every PR): Recall@k on 50-question golden dataset, threshold stability check, bulk insert completeness.

Tier 3 (nightly): Full golden dataset (500+ questions), distribution drift detection, multi-strategy comparison, performance benchmarks.

HelpMeTest can complement your vector database test suite by validating the user-facing impact — testing that real user queries on your application still return useful answers after index changes or embedding model updates, without requiring infrastructure setup or code.

Summary

Vector database testing is not optional. The failure modes are real, silent, and directly impact user experience. Cover all four areas:

  1. Embedding quality — semantic similarity tests, edge case handling, dimension validation
  2. Similarity calibration — threshold optimization via F1 maximization, regression detection
  3. Recall@k — golden dataset benchmark across k values, per-index-type validation
  4. Indexing correctness — bulk insert completeness, update and delete propagation, drift detection

Run the fast tests on every commit, integration tests on every PR, and the full evaluation nightly. Alert on any metric that drops more than 3-5 points from its baseline. Silent retrieval degradation is the most common and least noticed failure mode in production RAG systems.

Read more

Start now free