Testing Jupyter Notebooks with nbval and papermill: A Practical Guide
Jupyter notebooks are everywhere in data science and ML — for exploration, reporting, and increasingly for production pipelines. But they're notoriously hard to test. Cells run in any order, outputs depend on execution state, and "it worked on my machine" is the default failure mode.
Two tools solve this: nbval for regression testing (does the notebook still produce the same outputs?) and papermill for parametrization and execution (run the notebook programmatically with different inputs). This guide covers both, plus how to wire them into CI.
The Core Problem with Notebook Testing
Notebooks have two failure modes:
- Execution failures — a cell raises an exception
- Output drift — the notebook runs, but produces different results than expected
Most teams only catch failure mode 1 (if they test at all). nbval catches both.
Installing nbval and papermill
pip install nbval papermill pytest nbformatnbval: Regression Testing for Notebooks
nbval is a pytest plugin that re-executes a notebook and compares cell outputs to stored outputs. If any cell produces different output, the test fails.
Basic Usage
Run nbval on a notebook that has already been executed (with outputs saved):
# First, execute the notebook and save outputs
jupyter nbconvert --to notebook --execute my_analysis.ipynb --output my_analysis_executed.ipynb
# Then run nbval
pytest --nbval my_analysis_executed.ipynb -vOr use --nbval-lax to only check for execution errors (not output differences):
pytest --nbval-lax my_analysis.ipynb -vChecking Specific Notebooks in CI
The typical CI workflow:
# .github/workflows/notebooks.yml
name: Notebook Tests
on: [push, pull_request]
jobs:
test-notebooks:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.12" }
- run: pip install nbval papermill pytest jupyter pandas numpy scikit-learn
- name: Test notebooks execute without errors
run: pytest --nbval-lax notebooks/ -v --timeout=300--nbval-lax is usually the right default for CI: it catches broken notebooks (cells that fail) without failing on minor output formatting differences (timestamps, memory addresses, random seeds).
Handling Non-Deterministic Outputs
Some cells produce non-deterministic outputs (timestamps, progress bars, memory addresses). Tell nbval to skip output comparison for those cells by adding a special tag:
In Jupyter: Edit → Cell Toolbar → Tags → Add nbval-skip
Or add it to the cell metadata directly in the .ipynb JSON:
{
"cell_type": "code",
"metadata": {
"tags": ["nbval-skip"]
},
"source": "print(f'Processed at {datetime.now()}')"
}For cells that produce output you want to check approximately (floats):
{
"metadata": {
"tags": ["nbval-check-output"]
}
}Sanitizing Output with nbval
For notebook cells that print floats or timestamps, use nbval's sanitize file to define regex replacements:
# sanitize.cfg
[regex1]
regex = \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}
replace = DATE_TIME
[regex2]
regex = \d+\.\d+s
replace = TIME_ELAPSEDpytest --nbval --sanitize-with sanitize.cfg my_notebook.ipynbpapermill: Parametrized Notebook Execution
papermill injects parameters into a notebook at runtime and executes it. This is useful for:
- Running the same analysis notebook for different date ranges or datasets
- Generating reports for multiple clients/segments
- Automating model training notebooks with different hyperparameters
- Testing notebooks with controlled inputs
Marking Parameter Cells
Tag the cell containing default parameters with parameters:
# In Jupyter, tag this cell with "parameters" using Cell → Tags
start_date = "2024-01-01"
end_date = "2024-01-31"
product_category = "electronics"
sample_size = 1000Running a Parametrized Notebook via CLI
papermill analysis.ipynb output/analysis_feb.ipynb \
-p start_date "2024-02-01" \
-p end_date "2024-02-29" \
-p product_category "electronics"Running a Parametrized Notebook via Python
import papermill as pm
pm.execute_notebook(
input_path="notebooks/monthly_report.ipynb",
output_path="output/report_march_2024.ipynb",
parameters={
"start_date": "2024-03-01",
"end_date": "2024-03-31",
"product_category": "electronics",
"sample_size": 5000,
},
kernel_name="python3",
)If any cell raises an exception, papermill raises papermill.exceptions.PapermillExecutionError.
Testing with papermill in pytest
# tests/test_notebooks.py
import pytest
import papermill as pm
from pathlib import Path
import tempfile
NOTEBOOKS_DIR = Path(__file__).parent.parent / "notebooks"
def test_monthly_report_notebook_executes():
"""Monthly report notebook should execute without errors."""
with tempfile.TemporaryDirectory() as tmpdir:
output_path = Path(tmpdir) / "test_output.ipynb"
pm.execute_notebook(
input_path=str(NOTEBOOKS_DIR / "monthly_report.ipynb"),
output_path=str(output_path),
parameters={
"start_date": "2024-01-01",
"end_date": "2024-01-07", # Small range for fast CI
"sample_size": 100,
},
)
assert output_path.exists()
@pytest.mark.parametrize("category", ["electronics", "clothing", "books"])
def test_report_notebook_all_categories(category):
"""Report should work for all product categories."""
with tempfile.TemporaryDirectory() as tmpdir:
output_path = Path(tmpdir) / f"test_{category}.ipynb"
pm.execute_notebook(
input_path=str(NOTEBOOKS_DIR / "monthly_report.ipynb"),
output_path=str(output_path),
parameters={
"start_date": "2024-01-01",
"end_date": "2024-01-07",
"product_category": category,
"sample_size": 50,
},
)
assert output_path.exists()Inspecting Notebook Output After Execution
papermill stores outputs in the executed notebook. You can read them back:
import nbformat
def get_notebook_outputs(notebook_path: str) -> dict:
"""Extract all cell outputs from an executed notebook."""
with open(notebook_path) as f:
nb = nbformat.read(f, as_version=4)
outputs = {}
for i, cell in enumerate(nb.cells):
if cell.cell_type == "code" and cell.outputs:
outputs[i] = [o.get("text", o.get("data", {}).get("text/plain", ""))
for o in cell.outputs]
return outputs
def test_report_notebook_generates_summary():
"""Report notebook should print a summary line."""
with tempfile.TemporaryDirectory() as tmpdir:
output_path = Path(tmpdir) / "output.ipynb"
pm.execute_notebook(
input_path=str(NOTEBOOKS_DIR / "monthly_report.ipynb"),
output_path=str(output_path),
parameters={"start_date": "2024-01-01", "end_date": "2024-01-07"},
)
outputs = get_notebook_outputs(str(output_path))
# Verify summary output exists in last cell
last_cell_outputs = outputs.get(len(outputs) - 1, [])
assert any("Total revenue" in str(o) for o in last_cell_outputs)Combining nbval and papermill
For production notebooks, use both:
- papermill to execute the notebook with controlled parameters
- nbval to compare the executed output against a known-good baseline
def test_notebook_output_regression():
"""Execute notebook and compare to baseline output."""
baseline = NOTEBOOKS_DIR / "baselines" / "monthly_report_baseline.ipynb"
with tempfile.TemporaryDirectory() as tmpdir:
output_path = Path(tmpdir) / "current.ipynb"
# Execute with same parameters as baseline
pm.execute_notebook(
input_path=str(NOTEBOOKS_DIR / "monthly_report.ipynb"),
output_path=str(output_path),
parameters={"start_date": "2024-01-01", "end_date": "2024-01-07", "sample_size": 100},
)
# Compare outputs with nbval-style comparison
# (simplified — in practice use nbval directly)
import subprocess
result = subprocess.run(
["pytest", "--nbval", "--sanitize-with", "sanitize.cfg", str(output_path)],
capture_output=True,
text=True,
)
assert result.returncode == 0, f"Notebook output regression:\n{result.stdout}"CI Workflow for Notebooks
A complete GitHub Actions workflow:
# .github/workflows/notebooks.yml
name: Notebook CI
on:
push:
paths:
- "notebooks/**"
- "src/**"
pull_request:
jobs:
execution-test:
name: Notebook Execution Tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.12" }
- name: Install dependencies
run: pip install -r requirements.txt nbval papermill pytest
- name: Test notebooks execute without errors (fast mode)
run: |
pytest --nbval-lax notebooks/ -v \
--timeout=120 \
--ignore=notebooks/exploratory/
- name: Run parametrized notebook tests
run: pytest tests/test_notebooks.py -v --timeout=300
regression-test:
name: Notebook Output Regression
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/main'
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.12" }
- run: pip install -r requirements.txt nbval papermill pytest
- name: Regression test against baselines
run: pytest --nbval notebooks/production/ -v --sanitize-with sanitize.cfgHandling Timeouts
Long-running cells will time out in CI. Set timeouts per notebook:
pytest --nbval-lax --timeout=60 notebooks/ -vOr use the NBVAL_CELL_TIMEOUT environment variable:
NBVAL_CELL_TIMEOUT=120 pytest --nbval-lax heavy_notebook.ipynbFor papermill:
pm.execute_notebook(
"long_running.ipynb",
"output.ipynb",
parameters={"sample_size": 100},
execution_timeout=120, # seconds per cell
)Best Practices
Keep CI notebooks small. Use small sample_size parameters for CI runs. Full-scale runs on nightly or release schedules.
Separate exploratory and production notebooks. Don't test notebooks/exploratory/ in CI — test only notebooks/production/ or notebooks/reports/.
Commit executed baseline notebooks. For output regression testing with nbval, commit the notebook with outputs. Update deliberately, not accidentally.
Skip volatile cells. Timestamp cells, progress bars, and print statements with environment info should be tagged nbval-skip.
Use --nbval-lax in PRs, --nbval on main. --nbval-lax catches broken notebooks fast. --nbval catches output regressions on merge.
Conclusion
nbval and papermill together give you the testing coverage notebooks need: execution testing (does it run without errors?), parametrization (does it work with different inputs?), and output regression testing (does it still produce the same results?). Wire these into GitHub Actions and you'll stop shipping broken notebooks to production.