Testing Time-Series ML Models: Backtesting, Walk-Forward Validation, and Forecasting Quality
Time-series ML models fail in ways that standard ML models don't. A model that looks great on cross-validation can be useless for forecasting — because random cross-validation violates the most fundamental rule of time series: the future cannot be used to predict the past.
This guide covers testing strategies specific to time-series models: how to split data correctly, which metrics matter, and how to build automated tests that catch temporal leakage and forecast degradation.
The Core Problem: Temporal Leakage
Random train/test splits are wrong for time series. If your dataset spans January–December and you randomly sample test points, some training samples will be from December while some test samples are from January. The model sees the future while training — inflated performance, useless model.
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
# WRONG — random split causes temporal leakage
df = pd.read_csv("sales_data.csv", parse_dates=["date"])
X_train, X_test = train_test_split(df, test_size=0.2, random_state=42)
# X_train contains data from December; X_test contains data from January
# Your model is effectively cheating
# RIGHT — chronological split
split_date = df["date"].quantile(0.8) # Use last 20% chronologically
X_train = df[df["date"] < split_date]
X_test = df[df["date"] >= split_date]Walk-Forward Validation
Walk-forward validation (also called time-series cross-validation) simulates real-world forecasting. You train on an expanding window, predict one step ahead, then advance the window.
from sklearn.model_selection import TimeSeriesSplit
import numpy as np
def walk_forward_validate(model_class, X, y, n_splits=5):
"""
Walk-forward validation for time-series models.
Each fold uses all past data and predicts the next block.
"""
tscv = TimeSeriesSplit(n_splits=n_splits)
fold_metrics = []
for fold_idx, (train_idx, test_idx) in enumerate(tscv.split(X)):
X_train, X_test = X.iloc[train_idx], X.iloc[test_idx]
y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]
model = model_class()
model.fit(X_train, y_train)
predictions = model.predict(X_test)
fold_metrics.append({
"fold": fold_idx,
"train_size": len(train_idx),
"test_size": len(test_idx),
"mae": mean_absolute_error(y_test, predictions),
"mape": mean_absolute_percentage_error(y_test, predictions),
"rmse": np.sqrt(mean_squared_error(y_test, predictions)),
})
return pd.DataFrame(fold_metrics)
def mean_absolute_percentage_error(y_true, y_pred):
return np.mean(np.abs((y_true - y_pred) / np.clip(np.abs(y_true), 1e-8, None)))Testing Forecasting Models
# tests/test_forecasting_model.py
import pytest
import pandas as pd
import numpy as np
from your_ml_project.forecasting import SalesForecaster
@pytest.fixture
def historical_sales():
"""Load historical sales data for testing."""
return pd.read_parquet("data/test/sales_history.parquet")
def test_forecast_horizon(historical_sales):
"""Model should forecast the requested number of periods."""
model = SalesForecaster()
model.fit(historical_sales)
forecast = model.predict(horizon=30) # 30-day forecast
assert len(forecast) == 30
assert forecast.index.is_monotonic_increasing
def test_no_negative_sales_forecast(historical_sales):
"""Sales forecast should never be negative."""
model = SalesForecaster()
model.fit(historical_sales)
forecast = model.predict(horizon=90)
assert (forecast["predicted_sales"] >= 0).all(), \
f"Negative sales in forecast: {forecast[forecast['predicted_sales'] < 0]}"
def test_forecast_confidence_intervals(historical_sales):
"""Confidence intervals must be valid."""
model = SalesForecaster()
model.fit(historical_sales)
forecast = model.predict(horizon=30, return_ci=True)
# Lower bound <= point estimate <= upper bound
assert (forecast["lower_80"] <= forecast["predicted"]).all()
assert (forecast["predicted"] <= forecast["upper_80"]).all()
# CI should widen over time (uncertainty increases)
ci_width = forecast["upper_80"] - forecast["lower_80"]
# First half should be tighter than second half on average
assert ci_width[:15].mean() <= ci_width[15:].mean()
def test_backtesting_accuracy_thresholds(historical_sales):
"""Model must meet accuracy thresholds in backtesting."""
# Use data up to 6 months ago for training, last 6 months for testing
cutoff = historical_sales.index.max() - pd.Timedelta(days=180)
train = historical_sales[:cutoff]
test = historical_sales[cutoff:]
model = SalesForecaster()
model.fit(train)
forecast = model.predict(horizon=len(test))
# MAPE threshold for this use case
mape = mean_absolute_percentage_error(test["sales"], forecast["predicted"])
assert mape <= 0.15, f"MAPE {mape:.1%} exceeds threshold of 15%"
# Coverage: 80% CI should contain 80% of actuals
in_ci = (
(test["sales"] >= forecast["lower_80"]) &
(test["sales"] <= forecast["upper_80"])
).mean()
assert in_ci >= 0.70, f"80% CI only covers {in_ci:.1%} of actuals"Testing for Temporal Leakage
Build automated tests that detect whether your features contain future information:
def test_no_temporal_leakage_in_features(feature_engineering_fn, raw_data):
"""Features must not contain information from the future."""
for test_date in pd.date_range(raw_data.index.min() + pd.Timedelta(days=30),
raw_data.index.max(),
freq='MS'): # Monthly checkpoints
# Get features as of test_date (what the model would see in production)
available_data = raw_data[:test_date]
features = feature_engineering_fn(available_data, as_of_date=test_date)
# Check that no feature values require future data to compute
# (implementation depends on your feature definitions)
assert features.index.max() <= test_date, \
f"Features computed for {features.index.max()} using data only up to {test_date}"
# Rolling features should use only past data
if "rolling_30d_avg" in features.columns:
# Verify rolling feature matches manual calculation
expected = raw_data["sales"][:test_date].rolling(30).mean()
pd.testing.assert_series_equal(
features["rolling_30d_avg"],
expected.reindex(features.index),
check_names=False,
)Seasonality Tests
def test_model_captures_weekly_seasonality(historical_sales, trained_forecaster):
"""Model should predict lower weekend sales in retail context."""
forecast = trained_forecaster.predict(horizon=14) # Two-week forecast
forecast["day_of_week"] = forecast.index.dayofweek
weekday_avg = forecast[forecast["day_of_week"] < 5]["predicted"].mean()
weekend_avg = forecast[forecast["day_of_week"] >= 5]["predicted"].mean()
# Weekday sales should be higher (domain knowledge for this dataset)
assert weekday_avg > weekend_avg * 0.9, \
f"Model doesn't capture weekday/weekend pattern: " \
f"weekday={weekday_avg:.0f}, weekend={weekend_avg:.0f}"
def test_model_captures_monthly_seasonality(historical_sales, trained_forecaster):
"""Model should predict seasonal patterns correctly."""
# Compute historical monthly factors
monthly_factors = (
historical_sales.groupby(historical_sales.index.month)["sales"].mean()
/ historical_sales["sales"].mean()
)
# Get next 12-month forecast
forecast = trained_forecaster.predict(horizon=365)
forecast_monthly = forecast.groupby(forecast.index.month)["predicted"].mean()
forecast_factors = forecast_monthly / forecast_monthly.mean()
# High-season months should still rank as high-season in forecast
high_season_months = monthly_factors.nlargest(3).index.tolist()
for month in high_season_months:
assert forecast_factors[month] >= 1.0, \
f"Month {month} should be above-average but forecast shows {forecast_factors[month]:.2f}x"Metrics That Matter for Forecasting
Not all metrics are equally useful for time-series evaluation:
import numpy as np
from typing import Dict
def compute_forecast_metrics(y_true: np.ndarray, y_pred: np.ndarray,
y_naive: np.ndarray = None) -> Dict[str, float]:
"""
Compute forecasting metrics.
y_naive: naive baseline (e.g., last-period carry-forward) for MASE.
"""
mae = np.mean(np.abs(y_true - y_pred))
mse = np.mean((y_true - y_pred) ** 2)
rmse = np.sqrt(mse)
# MAPE — interpretable but undefined when actuals are 0
nonzero = y_true != 0
mape = np.mean(np.abs((y_true[nonzero] - y_pred[nonzero]) / y_true[nonzero]))
# SMAPE — symmetric, handles zero actuals better
smape = np.mean(
2 * np.abs(y_true - y_pred) / (np.abs(y_true) + np.abs(y_pred) + 1e-8)
)
metrics = {"MAE": mae, "RMSE": rmse, "MAPE": mape, "SMAPE": smape}
# MASE — compares to naive baseline, scale-independent
if y_naive is not None:
naive_mae = np.mean(np.abs(y_true - y_naive))
mase = mae / (naive_mae + 1e-8)
metrics["MASE"] = mase
# MASE < 1 means you're beating the naive baseline
return metrics
def test_beats_naive_baseline(trained_forecaster, historical_sales):
"""Model must beat the naive last-value-carry-forward baseline."""
cutoff = historical_sales.index.max() - pd.Timedelta(days=30)
train = historical_sales[:cutoff]
test = historical_sales[cutoff:]
model = SalesForecaster()
model.fit(train)
model_forecast = model.predict(horizon=len(test))["predicted"].values
# Naive baseline: carry forward last observed value
naive_forecast = np.full(len(test), train["sales"].iloc[-1])
metrics = compute_forecast_metrics(
y_true=test["sales"].values,
y_pred=model_forecast,
y_naive=naive_forecast,
)
assert metrics["MASE"] < 1.0, \
f"Model MASE={metrics['MASE']:.2f} — not beating naive baseline"Drift Detection for Deployed Forecasters
In production, monitor for when your model's assumptions break down:
# monitoring/test_forecast_drift.py — run on schedule
import pytest
import pandas as pd
def test_recent_forecast_accuracy():
"""Compare last week's forecast to actual outcomes."""
week_ago = pd.Timestamp.now() - pd.Timedelta(days=7)
# Load forecasts made last week
past_forecasts = load_forecasts(made_at=week_ago)
# Load actual outcomes
actuals = load_actuals(period=past_forecasts.index)
if len(actuals) == 0:
pytest.skip("No actuals available yet for comparison")
mape = mean_absolute_percentage_error(actuals, past_forecasts)
# Alert if accuracy dropped significantly vs historical average
historical_mape = get_historical_mape(lookback_weeks=12)
assert mape <= historical_mape * 1.5, \
f"Recent MAPE {mape:.1%} is {mape/historical_mape:.1f}x worse than " \
f"historical average {historical_mape:.1%}"
def test_input_distribution_stable():
"""Input features should not have shifted dramatically."""
recent = load_feature_values(lookback_days=7)
baseline = load_feature_values(lookback_days=90, skip_recent_days=7)
from scipy import stats
for feature in ["traffic", "promotions_active", "weather_score"]:
_, p_value = stats.ks_2samp(baseline[feature], recent[feature])
assert p_value >= 0.01, \
f"Input feature '{feature}' distribution shifted significantly (p={p_value:.4f})"CI Integration
# .github/workflows/forecasting-tests.yml
name: Forecasting Model Tests
on:
push:
paths: ['models/**', 'src/forecasting/**', 'tests/forecasting/**']
schedule:
- cron: '0 5 * * *' # Daily accuracy monitoring
jobs:
backtesting:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run temporal leakage tests
run: pytest tests/forecasting/test_leakage.py -v
- name: Run walk-forward validation
run: pytest tests/forecasting/test_walk_forward.py -v
- name: Run accuracy thresholds
run: pytest tests/forecasting/test_accuracy.py -v
- name: Run seasonality tests
run: pytest tests/forecasting/test_seasonality.py -v
monitoring:
if: github.event_name == 'schedule'
runs-on: ubuntu-latest
steps:
- name: Run production accuracy monitoring
run: pytest tests/forecasting/test_production_accuracy.py -vCommon Time-Series Testing Mistakes
Mistake 1: Using r² for forecasting R² can be misleading for time-series — a persistent model (predict last value) can have very high R². Use MAE, MAPE, or MASE instead.
Mistake 2: Not testing the forecast horizon A model might be accurate for 7-day forecasts but terrible for 30-day forecasts. Test the horizon you actually use in production.
Mistake 3: Testing on the same seasonality period If your test set covers the same seasonal period as your training set, you're not testing generalization to new seasonal patterns. Make sure your test set includes at least one full seasonal cycle that doesn't appear in training.
Mistake 4: Ignoring prediction intervals A point forecast with no uncertainty estimate is often not useful. Test that your confidence intervals are calibrated — 80% intervals should contain ~80% of actuals.
Summary
Time-series testing requires different approaches than standard ML:
- Always use chronological splits — never random splits
- Walk-forward validation — simulates real production behavior
- Test for temporal leakage — catch future-data contamination
- Verify seasonality capture — model should learn domain patterns
- Test confidence interval calibration — not just point accuracy
- Beat the naive baseline — MASE < 1 is the minimum bar
- Monitor drift in production — distributions shift, models decay
Use HelpMeTest to run your backtesting and drift detection tests on a schedule. A forecasting model that was accurate three months ago might be degrading today — you need continuous testing to know.