Testing Feature Stores: Feast and Tecton Contract Testing and Drift Detection

Testing Feature Stores: Feast and Tecton Contract Testing and Drift Detection

Feature stores are one of the most undertested components in ML infrastructure. Teams spend weeks getting Feast or Tecton set up, then assume it works. But feature stores have unique failure modes that standard API testing misses: point-in-time leakage, offline/online store inconsistency, silent feature drift, and schema evolution bugs that only surface at training time.

This guide covers systematic testing for feature stores with practical examples for both Feast and Tecton.

What Can Go Wrong in Feature Stores

Before writing tests, map the actual failure modes:

Point-in-time leakage: Training data uses future features for past labels. The model trains and evaluates beautifully, then fails in production because the features it learned from won't exist at inference time.

Offline/online inconsistency: The offline store (Parquet/BigQuery) and online store (Redis/DynamoDB) drift apart. Training uses offline features; serving uses online features. If they differ, your training/serving skew is a bug, not a distribution shift.

Feature pipeline staleness: The materialization job fails silently. Your online store is serving features that are 48 hours stale. The model score degrades. Nobody notices until a monitoring alert fires.

Schema drift: A data source upstream changes a column type from int64 to float64, or renames a feature. Feast materializes successfully (it's a different column), and old feature requests return null silently.

Setting Up a Test Feature Store

Use Feast's local file-based provider for unit and integration tests:

# tests/fixtures/feature_store.py
import pytest
import tempfile
import os
from pathlib import Path
from feast import FeatureStore
from feast.repo_config import RepoConfig, RegistryConfig
from feast.infra.online_stores.sqlite import SqliteOnlineStoreConfig

@pytest.fixture(scope="function")
def local_feature_store(tmp_path):
    """Isolated Feast feature store for testing."""
    repo_path = tmp_path / "feature_repo"
    repo_path.mkdir()

    config = RepoConfig(
        registry=RegistryConfig(path=str(repo_path / "registry.db")),
        project="test_project",
        provider="local",
        online_store=SqliteOnlineStoreConfig(path=str(repo_path / "online_store.db")),
        offline_store={"type": "file"},
        entity_key_serialization_version=2,
    )

    store = FeatureStore(config=config)
    yield store

Define a minimal test feature repository:

# tests/fixtures/feature_definitions.py
from datetime import timedelta
import pandas as pd
from feast import (
    Entity, Feature, FeatureView, FileSource, ValueType, FeatureService
)
from feast.types import Float32, Int64, String

user_entity = Entity(
    name="user_id",
    value_type=ValueType.INT64,
    description="User identifier",
)

user_source = FileSource(
    path="tests/fixtures/user_features.parquet",
    timestamp_field="event_timestamp",
    created_timestamp_column="created",
)

user_features_view = FeatureView(
    name="user_features",
    entities=["user_id"],
    ttl=timedelta(days=30),
    schema=[
        Feature(name="age", dtype=Float32),
        Feature(name="account_balance", dtype=Float32),
        Feature(name="transaction_count_7d", dtype=Int64),
        Feature(name="country_code", dtype=String),
    ],
    online=True,
    source=user_source,
)

Contract Testing

Feature contracts define what downstream models expect. Test that your feature views deliver on those contracts.

# tests/test_feature_contracts.py
import pytest
import pandas as pd
import numpy as np
from datetime import datetime, timedelta
from feast import FeatureStore
from pydantic import BaseModel, validator
from typing import List, Dict, Any, Optional

class FeatureContract(BaseModel):
    """Defines what a model expects from a feature view."""
    feature_view: str
    required_features: List[str]
    feature_types: Dict[str, str]
    null_rate_threshold: float = 0.05  # Max 5% nulls allowed
    value_ranges: Dict[str, tuple] = {}

    def validate_dataframe(self, df: pd.DataFrame) -> None:
        # Check all required features present
        missing = set(self.required_features) - set(df.columns)
        if missing:
            raise AssertionError(f"Contract violation: missing features {missing}")

        # Check types
        for feature, expected_type in self.feature_types.items():
            if feature in df.columns:
                actual_dtype = str(df[feature].dtype)
                if expected_type not in actual_dtype:
                    raise AssertionError(
                        f"Feature '{feature}' type mismatch: "
                        f"expected {expected_type}, got {actual_dtype}"
                    )

        # Check null rates
        for feature in self.required_features:
            if feature in df.columns:
                null_rate = df[feature].isnull().mean()
                if null_rate > self.null_rate_threshold:
                    raise AssertionError(
                        f"Feature '{feature}' null rate {null_rate:.3f} "
                        f"exceeds threshold {self.null_rate_threshold}"
                    )

        # Check value ranges
        for feature, (min_val, max_val) in self.value_ranges.items():
            if feature in df.columns:
                col = df[feature].dropna()
                if len(col) > 0:
                    if col.min() < min_val or col.max() > max_val:
                        raise AssertionError(
                            f"Feature '{feature}' values out of range [{min_val}, {max_val}]: "
                            f"actual [{col.min()}, {col.max()}]"
                        )

CHURN_MODEL_CONTRACT = FeatureContract(
    feature_view="user_features",
    required_features=["age", "account_balance", "transaction_count_7d", "country_code"],
    feature_types={
        "age": "float",
        "account_balance": "float",
        "transaction_count_7d": "int",
    },
    value_ranges={
        "age": (0, 150),
        "account_balance": (-1e6, 1e9),
        "transaction_count_7d": (0, 10000),
    },
)

class TestFeatureContracts:
    def test_user_features_satisfy_churn_model_contract(self, local_feature_store):
        """Verify the user feature view satisfies the churn model's contract."""
        # Create test data
        now = pd.Timestamp.now(tz="UTC")
        df = pd.DataFrame({
            "user_id": [1, 2, 3, 4, 5],
            "age": [25.0, 34.0, 45.0, 28.0, 52.0],
            "account_balance": [1000.0, 5000.0, 250.0, 3000.0, 12000.0],
            "transaction_count_7d": [3, 12, 1, 8, 5],
            "country_code": ["US", "UK", "DE", "US", "FR"],
            "event_timestamp": [now - timedelta(hours=i) for i in range(5)],
            "created": [now] * 5,
        })

        CHURN_MODEL_CONTRACT.validate_dataframe(df)  # Should not raise

    def test_contract_fails_on_missing_feature(self):
        df = pd.DataFrame({
            "age": [25.0, 34.0],
            # Missing account_balance, transaction_count_7d, country_code
        })
        with pytest.raises(AssertionError, match="missing features"):
            CHURN_MODEL_CONTRACT.validate_dataframe(df)

    def test_contract_fails_on_high_null_rate(self):
        df = pd.DataFrame({
            "age": [None] * 10 + [25.0],  # 91% nulls
            "account_balance": [1000.0] * 11,
            "transaction_count_7d": [3] * 11,
            "country_code": ["US"] * 11,
        })
        with pytest.raises(AssertionError, match="null rate"):
            CHURN_MODEL_CONTRACT.validate_dataframe(df)

    def test_contract_fails_on_out_of_range_values(self):
        df = pd.DataFrame({
            "age": [200.0, 25.0],  # 200 years old is invalid
            "account_balance": [1000.0, 5000.0],
            "transaction_count_7d": [3, 12],
            "country_code": ["US", "UK"],
        })
        with pytest.raises(AssertionError, match="out of range"):
            CHURN_MODEL_CONTRACT.validate_dataframe(df)

Point-in-Time Correctness Tests

This is the most important test category for feature stores. Point-in-time joins must not leak future data into training.

# tests/test_point_in_time_correctness.py
import pandas as pd
import numpy as np
from datetime import datetime, timedelta
import pytest

def create_feature_history(user_id: int) -> pd.DataFrame:
    """Create a feature history with known values at known timestamps."""
    base_time = datetime(2024, 1, 1, tzinfo=pd.Timestamp.now(tz="UTC").tzinfo)
    return pd.DataFrame({
        "user_id": [user_id] * 5,
        "account_balance": [100.0, 200.0, 300.0, 400.0, 500.0],
        "event_timestamp": [
            base_time + timedelta(days=i) for i in range(5)
        ],
        "created": [base_time] * 5,
    })

class TestPointInTimeCorrectness:
    def test_no_future_feature_leakage(self, local_feature_store, tmp_path):
        """
        Given features at t=0,1,2,3,4, a training entity at t=2
        should get the feature value from t=2, NOT t=3 or t=4.
        """
        feature_data = create_feature_history(user_id=42)
        parquet_path = tmp_path / "user_features.parquet"
        feature_data.to_parquet(parquet_path)

        base_time = datetime(2024, 1, 1, tzinfo=pd.Timestamp.now(tz="UTC").tzinfo)
        # Entity timestamp is at day 2 — should get balance=300.0 (day 2 value)
        entity_df = pd.DataFrame({
            "user_id": [42],
            "event_timestamp": [base_time + timedelta(days=2)],
        })

        # If we had balance=400.0 or 500.0, we have leakage
        # The test documents the expected behavior; implementation varies by backend
        # Here we test the logic that performs the join
        joined = pd.merge_asof(
            entity_df.sort_values("event_timestamp"),
            feature_data[["user_id", "account_balance", "event_timestamp"]]
                .sort_values("event_timestamp"),
            on="event_timestamp",
            by="user_id",
            direction="backward",  # "backward" = no future leakage
        )

        assert joined["account_balance"].iloc[0] == 300.0, (
            f"Expected 300.0 (day 2 value), got {joined['account_balance'].iloc[0]} — "
            "this indicates future feature leakage"
        )

    def test_forward_join_would_cause_leakage(self):
        """Demonstrate that direction='forward' leaks future data — negative test."""
        base_time = datetime(2024, 1, 1, tzinfo=pd.Timestamp.now(tz="UTC").tzinfo)
        feature_data = create_feature_history(user_id=42)
        entity_df = pd.DataFrame({
            "user_id": [42],
            "event_timestamp": [base_time + timedelta(days=2)],
        })

        # direction='forward' looks for the NEXT feature value after entity timestamp
        joined = pd.merge_asof(
            entity_df.sort_values("event_timestamp"),
            feature_data[["user_id", "account_balance", "event_timestamp"]]
                .sort_values("event_timestamp"),
            on="event_timestamp",
            by="user_id",
            direction="forward",
        )

        # This returns day 2's exact value (300.0) or day 3 (400.0) depending on exact match
        # The point: direction='forward' must NEVER be used for training data joins
        # This test documents the dangerous behavior
        assert joined["account_balance"].iloc[0] >= 300.0

    def test_ttl_respects_feature_freshness(self, local_feature_store):
        """Features older than TTL should not be returned."""
        base_time = datetime(2024, 1, 1, tzinfo=pd.Timestamp.now(tz="UTC").tzinfo)
        feature_data = pd.DataFrame({
            "user_id": [99],
            "account_balance": [999.0],
            "event_timestamp": [base_time],  # Very old
            "created": [base_time],
        })

        entity_df = pd.DataFrame({
            "user_id": [99],
            "event_timestamp": [base_time + timedelta(days=60)],  # 60 days later
        })

        # With TTL=30 days, the 60-day-old feature should be treated as missing
        ttl_days = 30
        joined = pd.merge_asof(
            entity_df.sort_values("event_timestamp"),
            feature_data.sort_values("event_timestamp"),
            on="event_timestamp",
            by="user_id",
            direction="backward",
            tolerance=pd.Timedelta(days=ttl_days),
        )

        assert pd.isnull(joined["account_balance"].iloc[0]), (
            "Feature older than TTL should return null"
        )

Offline/Online Store Consistency Tests

The most operationally dangerous bug: offline and online stores return different values for the same entity.

# tests/test_offline_online_consistency.py
import pytest
import pandas as pd
import numpy as np
from datetime import datetime

class TestOfflineOnlineConsistency:
    def test_online_matches_offline_for_recent_entities(self, local_feature_store):
        """
        After materialization, online store should return same values as offline store
        for the same entity at the same point in time.
        """
        now = pd.Timestamp.now(tz="UTC")

        # Write known data to offline store
        test_data = pd.DataFrame({
            "user_id": [1, 2, 3],
            "account_balance": [1000.0, 2000.0, 3000.0],
            "transaction_count_7d": [5, 10, 15],
            "event_timestamp": [now] * 3,
            "created": [now] * 3,
        })

        # Materialize to online store
        local_feature_store.materialize_incremental(end_date=now.to_pydatetime())

        # Fetch from online store
        online_features = local_feature_store.get_online_features(
            features=["user_features:account_balance", "user_features:transaction_count_7d"],
            entity_rows=[{"user_id": 1}, {"user_id": 2}, {"user_id": 3}],
        ).to_df()

        # Fetch from offline store (point-in-time)
        entity_df = pd.DataFrame({
            "user_id": [1, 2, 3],
            "event_timestamp": [now] * 3,
        })
        offline_features = local_feature_store.get_historical_features(
            entity_df=entity_df,
            features=["user_features:account_balance", "user_features:transaction_count_7d"],
        ).to_df()

        # Compare — allow small float tolerance
        for user_id in [1, 2, 3]:
            online_balance = online_features[
                online_features["user_id"] == user_id
            ]["user_features__account_balance"].iloc[0]
            offline_balance = offline_features[
                offline_features["user_id"] == user_id
            ]["user_features__account_balance"].iloc[0]

            np.testing.assert_allclose(
                online_balance, offline_balance, rtol=1e-5,
                err_msg=f"Online/offline mismatch for user {user_id}: "
                        f"online={online_balance}, offline={offline_balance}"
            )

    def test_missing_entity_returns_null_not_error(self, local_feature_store):
        """Online store should return null for unknown entities, not raise an exception."""
        features = local_feature_store.get_online_features(
            features=["user_features:account_balance"],
            entity_rows=[{"user_id": 999999}],  # Does not exist
        ).to_df()

        assert pd.isnull(features["user_features__account_balance"].iloc[0])

Feature Drift Detection

Catch statistical drift before it affects model performance:

# tests/test_feature_drift.py
from scipy import stats
import numpy as np
import pandas as pd

def detect_feature_drift(
    reference_df: pd.DataFrame,
    current_df: pd.DataFrame,
    feature_cols: list,
    ks_threshold: float = 0.05,
) -> dict:
    """
    Kolmogorov-Smirnov test for distribution shift.
    Returns {feature: p_value} for each feature.
    """
    results = {}
    for col in feature_cols:
        ref = reference_df[col].dropna().values
        cur = current_df[col].dropna().values
        if len(ref) == 0 or len(cur) == 0:
            results[col] = {"p_value": 0.0, "drifted": True, "reason": "empty"}
            continue
        ks_stat, p_value = stats.ks_2samp(ref, cur)
        results[col] = {
            "p_value": p_value,
            "ks_statistic": ks_stat,
            "drifted": p_value < ks_threshold,
        }
    return results

class TestFeatureDrift:
    def test_identical_distributions_no_drift(self):
        np.random.seed(42)
        data = np.random.normal(0, 1, 1000)
        ref = pd.DataFrame({"balance": data[:500]})
        cur = pd.DataFrame({"balance": data[500:]})
        results = detect_feature_drift(ref, cur, ["balance"])
        assert not results["balance"]["drifted"], (
            "Same distribution should not be flagged as drifted"
        )

    def test_shifted_mean_detected_as_drift(self):
        np.random.seed(42)
        ref = pd.DataFrame({"balance": np.random.normal(0, 1, 1000)})
        cur = pd.DataFrame({"balance": np.random.normal(5, 1, 1000)})  # Mean shifted by 5
        results = detect_feature_drift(ref, cur, ["balance"])
        assert results["balance"]["drifted"], (
            "Mean shift of 5 sigma should be detected as drift"
        )

    def test_returns_per_feature_results(self):
        np.random.seed(0)
        ref = pd.DataFrame({
            "age": np.random.normal(35, 10, 500),
            "balance": np.random.normal(1000, 500, 500),
        })
        cur = pd.DataFrame({
            "age": np.random.normal(35, 10, 500),  # No drift
            "balance": np.random.normal(2000, 500, 500),  # Drifted
        })
        results = detect_feature_drift(ref, cur, ["age", "balance"])
        assert not results["age"]["drifted"]
        assert results["balance"]["drifted"]

CI Integration for Feature Store Tests

# .github/workflows/feature-store-tests.yaml
name: Feature Store Tests
on:
  push:
    paths:
      - "feature_repo/**"
      - "tests/feature_store/**"

jobs:
  feature-store-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - name: Install dependencies
        run: pip install feast==0.36.0 pytest pandas scipy numpy pydantic
      - name: Run contract tests
        run: pytest tests/feature_store/test_contracts.py -v
      - name: Run point-in-time correctness tests
        run: pytest tests/feature_store/test_point_in_time.py -v
      - name: Run offline/online consistency tests
        run: pytest tests/feature_store/test_consistency.py -v
      - name: Run drift detection tests
        run: pytest tests/feature_store/test_drift.py -v

Summary

The most critical tests for feature stores:

  1. Contract tests — encode what each model needs (feature names, types, null rates, value ranges) as a FeatureContract and test against it every pipeline run.
  2. Point-in-time correctness — test that training joins use direction='backward' and that TTL is respected. One wrong parameter causes silent leakage.
  3. Offline/online consistency — after every materialization, spot-check that online and offline values match for a sample of entities.
  4. Drift detection — run KS tests comparing today's feature distribution to a reference baseline; fail CI when p-value drops below threshold.
  5. Missing entity behavior — verify your online store returns null (not an error) for unknown entities.

For end-to-end monitoring of feature store pipelines in production — including alerting on materialization staleness and consistency regressions — HelpMeTest can run these checks on a schedule without custom monitoring infrastructure.

Read more

Start now free