Testing Jupyter Notebooks with nbval and papermill: A Practical Guide

Testing Jupyter Notebooks with nbval and papermill: A Practical Guide

Jupyter notebooks are everywhere in data science and ML — for exploration, reporting, and increasingly for production pipelines. But they're notoriously hard to test. Cells run in any order, outputs depend on execution state, and "it worked on my machine" is the default failure mode.

Two tools solve this: nbval for regression testing (does the notebook still produce the same outputs?) and papermill for parametrization and execution (run the notebook programmatically with different inputs). This guide covers both, plus how to wire them into CI.

The Core Problem with Notebook Testing

Notebooks have two failure modes:

  1. Execution failures — a cell raises an exception
  2. Output drift — the notebook runs, but produces different results than expected

Most teams only catch failure mode 1 (if they test at all). nbval catches both.

Installing nbval and papermill

pip install nbval papermill pytest nbformat

nbval: Regression Testing for Notebooks

nbval is a pytest plugin that re-executes a notebook and compares cell outputs to stored outputs. If any cell produces different output, the test fails.

Basic Usage

Run nbval on a notebook that has already been executed (with outputs saved):

# First, execute the notebook and save outputs
jupyter nbconvert --to notebook --execute my_analysis.ipynb --output my_analysis_executed.ipynb

# Then run nbval
pytest --nbval my_analysis_executed.ipynb -v

Or use --nbval-lax to only check for execution errors (not output differences):

pytest --nbval-lax my_analysis.ipynb -v

Checking Specific Notebooks in CI

The typical CI workflow:

# .github/workflows/notebooks.yml
name: Notebook Tests

on: [push, pull_request]

jobs:
  test-notebooks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install nbval papermill pytest jupyter pandas numpy scikit-learn
      - name: Test notebooks execute without errors
        run: pytest --nbval-lax notebooks/ -v --timeout=300

--nbval-lax is usually the right default for CI: it catches broken notebooks (cells that fail) without failing on minor output formatting differences (timestamps, memory addresses, random seeds).

Handling Non-Deterministic Outputs

Some cells produce non-deterministic outputs (timestamps, progress bars, memory addresses). Tell nbval to skip output comparison for those cells by adding a special tag:

In Jupyter: Edit → Cell Toolbar → Tags → Add nbval-skip

Or add it to the cell metadata directly in the .ipynb JSON:

{
  "cell_type": "code",
  "metadata": {
    "tags": ["nbval-skip"]
  },
  "source": "print(f'Processed at {datetime.now()}')"
}

For cells that produce output you want to check approximately (floats):

{
  "metadata": {
    "tags": ["nbval-check-output"]
  }
}

Sanitizing Output with nbval

For notebook cells that print floats or timestamps, use nbval's sanitize file to define regex replacements:

# sanitize.cfg
[regex1]
regex = \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}
replace = DATE_TIME

[regex2]
regex = \d+\.\d+s
replace = TIME_ELAPSED
pytest --nbval --sanitize-with sanitize.cfg my_notebook.ipynb

papermill: Parametrized Notebook Execution

papermill injects parameters into a notebook at runtime and executes it. This is useful for:

  • Running the same analysis notebook for different date ranges or datasets
  • Generating reports for multiple clients/segments
  • Automating model training notebooks with different hyperparameters
  • Testing notebooks with controlled inputs

Marking Parameter Cells

Tag the cell containing default parameters with parameters:

# In Jupyter, tag this cell with "parameters" using Cell → Tags
start_date = "2024-01-01"
end_date = "2024-01-31"
product_category = "electronics"
sample_size = 1000

Running a Parametrized Notebook via CLI

papermill analysis.ipynb output/analysis_feb.ipynb \
  -p start_date "2024-02-01" \
  -p end_date "2024-02-29" \
  -p product_category "electronics"

Running a Parametrized Notebook via Python

import papermill as pm

pm.execute_notebook(
    input_path="notebooks/monthly_report.ipynb",
    output_path="output/report_march_2024.ipynb",
    parameters={
        "start_date": "2024-03-01",
        "end_date": "2024-03-31",
        "product_category": "electronics",
        "sample_size": 5000,
    },
    kernel_name="python3",
)

If any cell raises an exception, papermill raises papermill.exceptions.PapermillExecutionError.

Testing with papermill in pytest

# tests/test_notebooks.py
import pytest
import papermill as pm
from pathlib import Path
import tempfile

NOTEBOOKS_DIR = Path(__file__).parent.parent / "notebooks"

def test_monthly_report_notebook_executes():
    """Monthly report notebook should execute without errors."""
    with tempfile.TemporaryDirectory() as tmpdir:
        output_path = Path(tmpdir) / "test_output.ipynb"
        
        pm.execute_notebook(
            input_path=str(NOTEBOOKS_DIR / "monthly_report.ipynb"),
            output_path=str(output_path),
            parameters={
                "start_date": "2024-01-01",
                "end_date": "2024-01-07",  # Small range for fast CI
                "sample_size": 100,
            },
        )
        
        assert output_path.exists()

@pytest.mark.parametrize("category", ["electronics", "clothing", "books"])
def test_report_notebook_all_categories(category):
    """Report should work for all product categories."""
    with tempfile.TemporaryDirectory() as tmpdir:
        output_path = Path(tmpdir) / f"test_{category}.ipynb"
        
        pm.execute_notebook(
            input_path=str(NOTEBOOKS_DIR / "monthly_report.ipynb"),
            output_path=str(output_path),
            parameters={
                "start_date": "2024-01-01",
                "end_date": "2024-01-07",
                "product_category": category,
                "sample_size": 50,
            },
        )
        
        assert output_path.exists()

Inspecting Notebook Output After Execution

papermill stores outputs in the executed notebook. You can read them back:

import nbformat

def get_notebook_outputs(notebook_path: str) -> dict:
    """Extract all cell outputs from an executed notebook."""
    with open(notebook_path) as f:
        nb = nbformat.read(f, as_version=4)
    
    outputs = {}
    for i, cell in enumerate(nb.cells):
        if cell.cell_type == "code" and cell.outputs:
            outputs[i] = [o.get("text", o.get("data", {}).get("text/plain", "")) 
                         for o in cell.outputs]
    return outputs

def test_report_notebook_generates_summary():
    """Report notebook should print a summary line."""
    with tempfile.TemporaryDirectory() as tmpdir:
        output_path = Path(tmpdir) / "output.ipynb"
        
        pm.execute_notebook(
            input_path=str(NOTEBOOKS_DIR / "monthly_report.ipynb"),
            output_path=str(output_path),
            parameters={"start_date": "2024-01-01", "end_date": "2024-01-07"},
        )
        
        outputs = get_notebook_outputs(str(output_path))
        
        # Verify summary output exists in last cell
        last_cell_outputs = outputs.get(len(outputs) - 1, [])
        assert any("Total revenue" in str(o) for o in last_cell_outputs)

Combining nbval and papermill

For production notebooks, use both:

  1. papermill to execute the notebook with controlled parameters
  2. nbval to compare the executed output against a known-good baseline
def test_notebook_output_regression():
    """Execute notebook and compare to baseline output."""
    baseline = NOTEBOOKS_DIR / "baselines" / "monthly_report_baseline.ipynb"
    
    with tempfile.TemporaryDirectory() as tmpdir:
        output_path = Path(tmpdir) / "current.ipynb"
        
        # Execute with same parameters as baseline
        pm.execute_notebook(
            input_path=str(NOTEBOOKS_DIR / "monthly_report.ipynb"),
            output_path=str(output_path),
            parameters={"start_date": "2024-01-01", "end_date": "2024-01-07", "sample_size": 100},
        )
        
        # Compare outputs with nbval-style comparison
        # (simplified — in practice use nbval directly)
        import subprocess
        result = subprocess.run(
            ["pytest", "--nbval", "--sanitize-with", "sanitize.cfg", str(output_path)],
            capture_output=True,
            text=True,
        )
        assert result.returncode == 0, f"Notebook output regression:\n{result.stdout}"

CI Workflow for Notebooks

A complete GitHub Actions workflow:

# .github/workflows/notebooks.yml
name: Notebook CI

on:
  push:
    paths:
      - "notebooks/**"
      - "src/**"
  pull_request:

jobs:
  execution-test:
    name: Notebook Execution Tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      
      - name: Install dependencies
        run: pip install -r requirements.txt nbval papermill pytest
      
      - name: Test notebooks execute without errors (fast mode)
        run: |
          pytest --nbval-lax notebooks/ -v \
            --timeout=120 \
            --ignore=notebooks/exploratory/
      
      - name: Run parametrized notebook tests
        run: pytest tests/test_notebooks.py -v --timeout=300

  regression-test:
    name: Notebook Output Regression
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements.txt nbval papermill pytest
      - name: Regression test against baselines
        run: pytest --nbval notebooks/production/ -v --sanitize-with sanitize.cfg

Handling Timeouts

Long-running cells will time out in CI. Set timeouts per notebook:

pytest --nbval-lax --timeout=60 notebooks/ -v

Or use the NBVAL_CELL_TIMEOUT environment variable:

NBVAL_CELL_TIMEOUT=120 pytest --nbval-lax heavy_notebook.ipynb

For papermill:

pm.execute_notebook(
    "long_running.ipynb",
    "output.ipynb",
    parameters={"sample_size": 100},
    execution_timeout=120,  # seconds per cell
)

Best Practices

Keep CI notebooks small. Use small sample_size parameters for CI runs. Full-scale runs on nightly or release schedules.

Separate exploratory and production notebooks. Don't test notebooks/exploratory/ in CI — test only notebooks/production/ or notebooks/reports/.

Commit executed baseline notebooks. For output regression testing with nbval, commit the notebook with outputs. Update deliberately, not accidentally.

Skip volatile cells. Timestamp cells, progress bars, and print statements with environment info should be tagged nbval-skip.

Use --nbval-lax in PRs, --nbval on main. --nbval-lax catches broken notebooks fast. --nbval catches output regressions on merge.

Conclusion

nbval and papermill together give you the testing coverage notebooks need: execution testing (does it run without errors?), parametrization (does it work with different inputs?), and output regression testing (does it still produce the same results?). Wire these into GitHub Actions and you'll stop shipping broken notebooks to production.

Read more

Start now free