Testing Time-Series ML Models: Backtesting, Walk-Forward Validation, and Forecasting Quality

Testing Time-Series ML Models: Backtesting, Walk-Forward Validation, and Forecasting Quality

Time-series ML models fail in ways that standard ML models don't. A model that looks great on cross-validation can be useless for forecasting — because random cross-validation violates the most fundamental rule of time series: the future cannot be used to predict the past.

This guide covers testing strategies specific to time-series models: how to split data correctly, which metrics matter, and how to build automated tests that catch temporal leakage and forecast degradation.

The Core Problem: Temporal Leakage

Random train/test splits are wrong for time series. If your dataset spans January–December and you randomly sample test points, some training samples will be from December while some test samples are from January. The model sees the future while training — inflated performance, useless model.

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split

# WRONG — random split causes temporal leakage
df = pd.read_csv("sales_data.csv", parse_dates=["date"])
X_train, X_test = train_test_split(df, test_size=0.2, random_state=42)
# X_train contains data from December; X_test contains data from January
# Your model is effectively cheating

# RIGHT — chronological split
split_date = df["date"].quantile(0.8)  # Use last 20% chronologically
X_train = df[df["date"] < split_date]
X_test = df[df["date"] >= split_date]

Walk-Forward Validation

Walk-forward validation (also called time-series cross-validation) simulates real-world forecasting. You train on an expanding window, predict one step ahead, then advance the window.

from sklearn.model_selection import TimeSeriesSplit
import numpy as np

def walk_forward_validate(model_class, X, y, n_splits=5):
    """
    Walk-forward validation for time-series models.
    Each fold uses all past data and predicts the next block.
    """
    tscv = TimeSeriesSplit(n_splits=n_splits)
    fold_metrics = []
    
    for fold_idx, (train_idx, test_idx) in enumerate(tscv.split(X)):
        X_train, X_test = X.iloc[train_idx], X.iloc[test_idx]
        y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]
        
        model = model_class()
        model.fit(X_train, y_train)
        predictions = model.predict(X_test)
        
        fold_metrics.append({
            "fold": fold_idx,
            "train_size": len(train_idx),
            "test_size": len(test_idx),
            "mae": mean_absolute_error(y_test, predictions),
            "mape": mean_absolute_percentage_error(y_test, predictions),
            "rmse": np.sqrt(mean_squared_error(y_test, predictions)),
        })
    
    return pd.DataFrame(fold_metrics)

def mean_absolute_percentage_error(y_true, y_pred):
    return np.mean(np.abs((y_true - y_pred) / np.clip(np.abs(y_true), 1e-8, None)))

Testing Forecasting Models

# tests/test_forecasting_model.py
import pytest
import pandas as pd
import numpy as np
from your_ml_project.forecasting import SalesForecaster

@pytest.fixture
def historical_sales():
    """Load historical sales data for testing."""
    return pd.read_parquet("data/test/sales_history.parquet")

def test_forecast_horizon(historical_sales):
    """Model should forecast the requested number of periods."""
    model = SalesForecaster()
    model.fit(historical_sales)
    
    forecast = model.predict(horizon=30)  # 30-day forecast
    
    assert len(forecast) == 30
    assert forecast.index.is_monotonic_increasing

def test_no_negative_sales_forecast(historical_sales):
    """Sales forecast should never be negative."""
    model = SalesForecaster()
    model.fit(historical_sales)
    
    forecast = model.predict(horizon=90)
    assert (forecast["predicted_sales"] >= 0).all(), \
        f"Negative sales in forecast: {forecast[forecast['predicted_sales'] < 0]}"

def test_forecast_confidence_intervals(historical_sales):
    """Confidence intervals must be valid."""
    model = SalesForecaster()
    model.fit(historical_sales)
    
    forecast = model.predict(horizon=30, return_ci=True)
    
    # Lower bound <= point estimate <= upper bound
    assert (forecast["lower_80"] <= forecast["predicted"]).all()
    assert (forecast["predicted"] <= forecast["upper_80"]).all()
    
    # CI should widen over time (uncertainty increases)
    ci_width = forecast["upper_80"] - forecast["lower_80"]
    # First half should be tighter than second half on average
    assert ci_width[:15].mean() <= ci_width[15:].mean()

def test_backtesting_accuracy_thresholds(historical_sales):
    """Model must meet accuracy thresholds in backtesting."""
    # Use data up to 6 months ago for training, last 6 months for testing
    cutoff = historical_sales.index.max() - pd.Timedelta(days=180)
    train = historical_sales[:cutoff]
    test = historical_sales[cutoff:]
    
    model = SalesForecaster()
    model.fit(train)
    forecast = model.predict(horizon=len(test))
    
    # MAPE threshold for this use case
    mape = mean_absolute_percentage_error(test["sales"], forecast["predicted"])
    assert mape <= 0.15, f"MAPE {mape:.1%} exceeds threshold of 15%"
    
    # Coverage: 80% CI should contain 80% of actuals
    in_ci = (
        (test["sales"] >= forecast["lower_80"]) & 
        (test["sales"] <= forecast["upper_80"])
    ).mean()
    assert in_ci >= 0.70, f"80% CI only covers {in_ci:.1%} of actuals"

Testing for Temporal Leakage

Build automated tests that detect whether your features contain future information:

def test_no_temporal_leakage_in_features(feature_engineering_fn, raw_data):
    """Features must not contain information from the future."""
    
    for test_date in pd.date_range(raw_data.index.min() + pd.Timedelta(days=30), 
                                    raw_data.index.max(), 
                                    freq='MS'):  # Monthly checkpoints
        
        # Get features as of test_date (what the model would see in production)
        available_data = raw_data[:test_date]
        features = feature_engineering_fn(available_data, as_of_date=test_date)
        
        # Check that no feature values require future data to compute
        # (implementation depends on your feature definitions)
        assert features.index.max() <= test_date, \
            f"Features computed for {features.index.max()} using data only up to {test_date}"
        
        # Rolling features should use only past data
        if "rolling_30d_avg" in features.columns:
            # Verify rolling feature matches manual calculation
            expected = raw_data["sales"][:test_date].rolling(30).mean()
            pd.testing.assert_series_equal(
                features["rolling_30d_avg"],
                expected.reindex(features.index),
                check_names=False,
            )

Seasonality Tests

def test_model_captures_weekly_seasonality(historical_sales, trained_forecaster):
    """Model should predict lower weekend sales in retail context."""
    forecast = trained_forecaster.predict(horizon=14)  # Two-week forecast
    forecast["day_of_week"] = forecast.index.dayofweek
    
    weekday_avg = forecast[forecast["day_of_week"] < 5]["predicted"].mean()
    weekend_avg = forecast[forecast["day_of_week"] >= 5]["predicted"].mean()
    
    # Weekday sales should be higher (domain knowledge for this dataset)
    assert weekday_avg > weekend_avg * 0.9, \
        f"Model doesn't capture weekday/weekend pattern: " \
        f"weekday={weekday_avg:.0f}, weekend={weekend_avg:.0f}"

def test_model_captures_monthly_seasonality(historical_sales, trained_forecaster):
    """Model should predict seasonal patterns correctly."""
    # Compute historical monthly factors
    monthly_factors = (
        historical_sales.groupby(historical_sales.index.month)["sales"].mean()
        / historical_sales["sales"].mean()
    )
    
    # Get next 12-month forecast
    forecast = trained_forecaster.predict(horizon=365)
    forecast_monthly = forecast.groupby(forecast.index.month)["predicted"].mean()
    forecast_factors = forecast_monthly / forecast_monthly.mean()
    
    # High-season months should still rank as high-season in forecast
    high_season_months = monthly_factors.nlargest(3).index.tolist()
    
    for month in high_season_months:
        assert forecast_factors[month] >= 1.0, \
            f"Month {month} should be above-average but forecast shows {forecast_factors[month]:.2f}x"

Metrics That Matter for Forecasting

Not all metrics are equally useful for time-series evaluation:

import numpy as np
from typing import Dict

def compute_forecast_metrics(y_true: np.ndarray, y_pred: np.ndarray, 
                              y_naive: np.ndarray = None) -> Dict[str, float]:
    """
    Compute forecasting metrics.
    y_naive: naive baseline (e.g., last-period carry-forward) for MASE.
    """
    mae = np.mean(np.abs(y_true - y_pred))
    mse = np.mean((y_true - y_pred) ** 2)
    rmse = np.sqrt(mse)
    
    # MAPE — interpretable but undefined when actuals are 0
    nonzero = y_true != 0
    mape = np.mean(np.abs((y_true[nonzero] - y_pred[nonzero]) / y_true[nonzero]))
    
    # SMAPE — symmetric, handles zero actuals better
    smape = np.mean(
        2 * np.abs(y_true - y_pred) / (np.abs(y_true) + np.abs(y_pred) + 1e-8)
    )
    
    metrics = {"MAE": mae, "RMSE": rmse, "MAPE": mape, "SMAPE": smape}
    
    # MASE — compares to naive baseline, scale-independent
    if y_naive is not None:
        naive_mae = np.mean(np.abs(y_true - y_naive))
        mase = mae / (naive_mae + 1e-8)
        metrics["MASE"] = mase
        # MASE < 1 means you're beating the naive baseline
    
    return metrics

def test_beats_naive_baseline(trained_forecaster, historical_sales):
    """Model must beat the naive last-value-carry-forward baseline."""
    cutoff = historical_sales.index.max() - pd.Timedelta(days=30)
    train = historical_sales[:cutoff]
    test = historical_sales[cutoff:]
    
    model = SalesForecaster()
    model.fit(train)
    model_forecast = model.predict(horizon=len(test))["predicted"].values
    
    # Naive baseline: carry forward last observed value
    naive_forecast = np.full(len(test), train["sales"].iloc[-1])
    
    metrics = compute_forecast_metrics(
        y_true=test["sales"].values,
        y_pred=model_forecast,
        y_naive=naive_forecast,
    )
    
    assert metrics["MASE"] < 1.0, \
        f"Model MASE={metrics['MASE']:.2f} — not beating naive baseline"

Drift Detection for Deployed Forecasters

In production, monitor for when your model's assumptions break down:

# monitoring/test_forecast_drift.py — run on schedule
import pytest
import pandas as pd

def test_recent_forecast_accuracy():
    """Compare last week's forecast to actual outcomes."""
    week_ago = pd.Timestamp.now() - pd.Timedelta(days=7)
    
    # Load forecasts made last week
    past_forecasts = load_forecasts(made_at=week_ago)
    
    # Load actual outcomes
    actuals = load_actuals(period=past_forecasts.index)
    
    if len(actuals) == 0:
        pytest.skip("No actuals available yet for comparison")
    
    mape = mean_absolute_percentage_error(actuals, past_forecasts)
    
    # Alert if accuracy dropped significantly vs historical average
    historical_mape = get_historical_mape(lookback_weeks=12)
    
    assert mape <= historical_mape * 1.5, \
        f"Recent MAPE {mape:.1%} is {mape/historical_mape:.1f}x worse than " \
        f"historical average {historical_mape:.1%}"

def test_input_distribution_stable():
    """Input features should not have shifted dramatically."""
    recent = load_feature_values(lookback_days=7)
    baseline = load_feature_values(lookback_days=90, skip_recent_days=7)
    
    from scipy import stats
    
    for feature in ["traffic", "promotions_active", "weather_score"]:
        _, p_value = stats.ks_2samp(baseline[feature], recent[feature])
        
        assert p_value >= 0.01, \
            f"Input feature '{feature}' distribution shifted significantly (p={p_value:.4f})"

CI Integration

# .github/workflows/forecasting-tests.yml
name: Forecasting Model Tests

on:
  push:
    paths: ['models/**', 'src/forecasting/**', 'tests/forecasting/**']
  schedule:
    - cron: '0 5 * * *'  # Daily accuracy monitoring

jobs:
  backtesting:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Run temporal leakage tests
        run: pytest tests/forecasting/test_leakage.py -v
      
      - name: Run walk-forward validation
        run: pytest tests/forecasting/test_walk_forward.py -v
      
      - name: Run accuracy thresholds
        run: pytest tests/forecasting/test_accuracy.py -v
      
      - name: Run seasonality tests
        run: pytest tests/forecasting/test_seasonality.py -v
  
  monitoring:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    steps:
      - name: Run production accuracy monitoring
        run: pytest tests/forecasting/test_production_accuracy.py -v

Common Time-Series Testing Mistakes

Mistake 1: Using r² for forecasting R² can be misleading for time-series — a persistent model (predict last value) can have very high R². Use MAE, MAPE, or MASE instead.

Mistake 2: Not testing the forecast horizon A model might be accurate for 7-day forecasts but terrible for 30-day forecasts. Test the horizon you actually use in production.

Mistake 3: Testing on the same seasonality period If your test set covers the same seasonal period as your training set, you're not testing generalization to new seasonal patterns. Make sure your test set includes at least one full seasonal cycle that doesn't appear in training.

Mistake 4: Ignoring prediction intervals A point forecast with no uncertainty estimate is often not useful. Test that your confidence intervals are calibrated — 80% intervals should contain ~80% of actuals.

Summary

Time-series testing requires different approaches than standard ML:

  1. Always use chronological splits — never random splits
  2. Walk-forward validation — simulates real production behavior
  3. Test for temporal leakage — catch future-data contamination
  4. Verify seasonality capture — model should learn domain patterns
  5. Test confidence interval calibration — not just point accuracy
  6. Beat the naive baseline — MASE < 1 is the minimum bar
  7. Monitor drift in production — distributions shift, models decay

Use HelpMeTest to run your backtesting and drift detection tests on a schedule. A forecasting model that was accurate three months ago might be degrading today — you need continuous testing to know.

Start now free