Testing

Testing Software Templates: Backstage, Cookiecutter, and Yeoman Scaffolding

HelpMeTest

HelpMeTest

— 8 min read
Testing Software Templates: Backstage, Cookiecutter, and Yeoman Scaffolding

Software templates and scaffolding tools bootstrap new projects, services, and components. When templates work correctly, developers get a fully wired-up starting point — CI/CD configured, linting set up, tests passing. When templates have bugs, every new project starts with broken infrastructure that nobody notices until it's too late.

This guide covers testing software templates across three common tools: Backstage Software Templates, Cookiecutter, and Yeoman.

Why Template Testing Matters

A broken template is worse than no template. Developers trust the scaffold to be correct, so they don't verify what it produced. Bugs hide for weeks until someone tries to actually deploy the service or run the tests.

Common template bugs:

  • Hardcoded values that should be templated (wrong project name in CI config)
  • Missing files (no .gitignore, no Dockerfile, no CI workflow)
  • Template syntax errors that silently produce wrong output
  • Generated code that doesn't compile or fails its own tests
  • Outdated dependencies pinned to vulnerable versions

Testing Backstage Software Templates

Backstage Software Templates use a YAML spec to define what gets created. They run template actions (fetch:template, publish:github, catalog:register) to produce output.

Template Schema Validation

# tests/backstage/test_template_schemas.py
import yaml
import glob
import pytest
from jsonschema import validate, ValidationError

# Simplified Backstage template schema (derive from official Backstage types)
TEMPLATE_SCHEMA = {
    "type": "object",
    "required": ["apiVersion", "kind", "metadata", "spec"],
    "properties": {
        "apiVersion": {"type": "string", "const": "scaffolder.backstage.io/v1beta3"},
        "kind": {"type": "string", "const": "Template"},
        "metadata": {
            "type": "object",
            "required": ["name", "title", "description"],
            "properties": {
                "name": {"type": "string", "pattern": "^[a-z][a-z0-9-]*$"},
                "title": {"type": "string"},
                "description": {"type": "string"},
                "tags": {"type": "array", "items": {"type": "string"}}
            }
        },
        "spec": {
            "type": "object",
            "required": ["owner", "type", "parameters", "steps"],
            "properties": {
                "owner": {"type": "string"},
                "type": {"type": "string"},
                "parameters": {"type": "array"},
                "steps": {"type": "array"}
            }
        }
    }
}

def load_template(path):
    with open(path) as f:
        return yaml.safe_load(f)

@pytest.mark.parametrize("path", glob.glob("backstage/templates/**/*.yaml", recursive=True))
def test_template_schema_valid(path):
    template = load_template(path)
    try:
        validate(template, TEMPLATE_SCHEMA)
    except ValidationError as e:
        pytest.fail(f"{path}: {e.message}")

@pytest.mark.parametrize("path", glob.glob("backstage/templates/**/*.yaml", recursive=True))
def test_template_has_description(path):
    template = load_template(path)
    desc = template.get("metadata", {}).get("description", "")
    assert len(desc) >= 30, f"{path}: description too short ({len(desc)} chars)"

@pytest.mark.parametrize("path", glob.glob("backstage/templates/**/*.yaml", recursive=True))
def test_template_steps_have_ids(path):
    template = load_template(path)
    steps = template.get("spec", {}).get("steps", [])
    
    for i, step in enumerate(steps):
        assert "id" in step, f"{path}: step {i} missing 'id'"
        assert "action" in step, f"{path}: step {i} missing 'action'"
        assert "name" in step, f"{path}: step {i} missing 'name'"

@pytest.mark.parametrize("path", glob.glob("backstage/templates/**/*.yaml", recursive=True))
def test_template_has_output(path):
    template = load_template(path)
    spec = template.get("spec", {})
    
    # Templates should define their outputs so catalog:register works
    assert "output" in spec, f"{path}: template missing 'output' section"
    links = spec["output"].get("links", [])
    assert len(links) > 0, f"{path}: template output has no links"

Template Content Validation

Verify that the Jinja/Nunjucks template files produce correct output.

# tests/backstage/test_template_content.py
from jinja2 import Environment, FileSystemLoader
import subprocess
import os
import shutil
import tempfile

def render_template_skeleton(template_name, params):
    """
    Render a Backstage template locally to test its output.
    This simulates what Backstage's fetch:template action does.
    """
    template_dir = f"backstage/templates/{template_name}/skeleton"
    output_dir = tempfile.mkdtemp()
    
    env = Environment(
        loader=FileSystemLoader(template_dir),
        # Backstage uses ${{ }} not {{ }}
        variable_start_string="${{",
        variable_end_string="}}",
        keep_trailing_newline=True
    )
    
    # Walk template files
    for root, dirs, files in os.walk(template_dir):
        for file in files:
            src_path = os.path.join(root, file)
            rel_path = os.path.relpath(src_path, template_dir)
            
            # Render filename (some templates have ${{ values.name }} in filenames)
            from jinja2 import Template
            rendered_rel = Template(
                rel_path,
                variable_start_string="${{",
                variable_end_string="}}"
            ).render(**params)
            
            dest_path = os.path.join(output_dir, rendered_rel)
            os.makedirs(os.path.dirname(dest_path), exist_ok=True)
            
            # Render file content
            try:
                template = env.get_template(rel_path)
                rendered = template.render(**params)
                with open(dest_path, "w") as f:
                    f.write(rendered)
            except Exception:
                # Binary file — copy as-is
                shutil.copy(src_path, dest_path)
    
    return output_dir

def test_python_service_template_produces_valid_python():
    """Generated Python service should have no syntax errors."""
    params = {
        "values": {
            "name": "my-service",
            "description": "A test service",
            "owner": "team-platform",
            "python_version": "3.12"
        }
    }
    
    output_dir = render_template_skeleton("python-service", params)
    
    try:
        # Check Python syntax
        result = subprocess.run(
            ["python", "-m", "py_compile"] + 
            [str(p) for p in Path(output_dir).rglob("*.py")],
            capture_output=True, text=True
        )
        assert result.returncode == 0, f"Syntax errors in generated Python:\n{result.stderr}"
        
        # Check that the service name appears correctly
        readme = Path(output_dir) / "README.md"
        assert readme.exists(), "README.md not generated"
        content = readme.read_text()
        assert "my-service" in content, "Service name not templated into README"
        assert "${{" not in content, "Unrendered template variables in README"
    
    finally:
        shutil.rmtree(output_dir)

def test_typescript_service_template_compiles():
    """Generated TypeScript service should compile without errors."""
    params = {
        "values": {
            "name": "my-ts-service",
            "description": "A TypeScript service",
            "owner": "team-backend",
            "node_version": "20"
        }
    }
    
    output_dir = render_template_skeleton("typescript-service", params)
    
    try:
        # Install dependencies and compile
        subprocess.run(["npm", "install"], cwd=output_dir, check=True, capture_output=True)
        
        result = subprocess.run(
            ["npx", "tsc", "--noEmit"],
            cwd=output_dir,
            capture_output=True, text=True
        )
        assert result.returncode == 0, f"TypeScript compilation failed:\n{result.stderr}"
    
    finally:
        shutil.rmtree(output_dir)

def test_generated_tests_pass():
    """The tests included in the template should pass out of the box."""
    params = {"values": {"name": "test-app", "owner": "team-qa"}}
    output_dir = render_template_skeleton("node-service", params)
    
    try:
        subprocess.run(["npm", "install"], cwd=output_dir, check=True, capture_output=True)
        
        result = subprocess.run(
            ["npm", "test"],
            cwd=output_dir,
            capture_output=True, text=True,
            timeout=120
        )
        assert result.returncode == 0, \
            f"Template's own tests failed:\n{result.stdout}\n{result.stderr}"
    
    finally:
        shutil.rmtree(output_dir)

Testing Cookiecutter Templates

Cookiecutter generates project scaffolds from Python templates.

pip install cookiecutter pytest

Basic Cookiecutter Tests

# tests/cookiecutter/test_template.py
import pytest
import subprocess
import shutil
import json
from pathlib import Path
import tempfile

TEMPLATE_DIR = Path("cookiecutter-python-service")

def bake(extra_context=None):
    """Run cookiecutter and return the output directory."""
    output_dir = tempfile.mkdtemp()
    
    context = {
        "project_name": "my-test-service",
        "project_slug": "my_test_service",
        "author": "Test Author",
        "email": "test@example.com",
        "python_version": "3.12",
        "use_docker": "yes",
        "use_ci": "yes"
    }
    if extra_context:
        context.update(extra_context)
    
    # Write context to a temp file
    context_file = Path(output_dir) / "context.json"
    context_file.write_text(json.dumps({"default_context": context}))
    
    result = subprocess.run(
        ["cookiecutter", str(TEMPLATE_DIR),
         "--no-input",
         "--config-file", str(context_file),
         "--output-dir", output_dir],
        capture_output=True, text=True
    )
    
    assert result.returncode == 0, f"Cookiecutter failed:\n{result.stderr}"
    
    # Find the generated project directory
    project_dirs = [d for d in Path(output_dir).iterdir() if d.is_dir()]
    assert len(project_dirs) == 1
    
    return project_dirs[0], output_dir

@pytest.fixture
def baked_project():
    project_dir, output_dir = bake()
    yield project_dir
    shutil.rmtree(output_dir)

def test_project_structure_correct(baked_project):
    """Generated project has all required files."""
    expected = [
        "README.md",
        "pyproject.toml",
        "Dockerfile",
        ".github/workflows/ci.yml",
        ".gitignore",
        "src/my_test_service/__init__.py",
        "tests/test_main.py"
    ]
    
    for path in expected:
        full_path = baked_project / path
        assert full_path.exists(), f"Missing expected file: {path}"

def test_project_name_substituted(baked_project):
    """Project name is correctly substituted throughout files."""
    readme = (baked_project / "README.md").read_text()
    assert "my-test-service" in readme
    assert "{{ cookiecutter.project_name }}" not in readme
    assert "{{cookiecutter" not in readme

def test_no_unrendered_templates(baked_project):
    """No files should contain unrendered Jinja2 variables."""
    for path in baked_project.rglob("*"):
        if path.is_file() and path.suffix in (".py", ".md", ".yaml", ".yml", ".toml", ".json"):
            content = path.read_text(errors="replace")
            assert "{{cookiecutter" not in content, \
                f"Unrendered template in {path.relative_to(baked_project)}"
            assert "{{ cookiecutter" not in content, \
                f"Unrendered template in {path.relative_to(baked_project)}"

def test_generated_python_valid_syntax(baked_project):
    """All generated Python files have valid syntax."""
    python_files = list(baked_project.rglob("*.py"))
    assert len(python_files) > 0, "No Python files generated"
    
    for py_file in python_files:
        result = subprocess.run(
            ["python", "-m", "py_compile", str(py_file)],
            capture_output=True, text=True
        )
        assert result.returncode == 0, \
            f"Syntax error in {py_file.name}:\n{result.stderr}"

def test_generated_tests_pass(baked_project):
    """Generated project's test suite passes."""
    # Install in a virtual environment
    venv_dir = baked_project / ".venv"
    subprocess.run(["python", "-m", "venv", str(venv_dir)], check=True)
    pip = venv_dir / "bin" / "pip"
    pytest_bin = venv_dir / "bin" / "pytest"
    
    subprocess.run([str(pip), "install", "-e", ".[dev]"],
                  cwd=baked_project, check=True, capture_output=True)
    
    result = subprocess.run(
        [str(pytest_bin), "tests/", "-v"],
        cwd=baked_project,
        capture_output=True, text=True,
        timeout=120
    )
    assert result.returncode == 0, \
        f"Generated tests failed:\n{result.stdout}\n{result.stderr}"

def test_docker_builds(baked_project):
    """Generated Dockerfile builds without errors."""
    result = subprocess.run(
        ["docker", "build", "-t", "test-baked-image:latest", "."],
        cwd=baked_project,
        capture_output=True, text=True,
        timeout=300
    )
    
    if result.returncode != 0:
        pytest.fail(f"Docker build failed:\n{result.stderr}")
    
    # Cleanup
    subprocess.run(["docker", "rmi", "test-baked-image:latest"], capture_output=True)

@pytest.mark.parametrize("project_slug", [
    "valid-slug",
    "another-valid-slug",
])
def test_valid_slugs_work(project_slug):
    """Various valid project slugs produce correct output."""
    project_dir, output_dir = bake(extra_context={
        "project_slug": project_slug.replace("-", "_"),
        "project_name": project_slug
    })
    try:
        readme = (project_dir / "README.md").read_text()
        assert project_slug in readme or project_slug.replace("-", "_") in readme
    finally:
        shutil.rmtree(output_dir)

Testing Yeoman Generators

Yeoman generators use a test helper package. Install it:

npm install --save-dev yeoman-test yeoman-assert chai

Generator Tests

// test/app.test.js
import assert from 'yeoman-assert';
import helpers from 'yeoman-test';
import path from 'path';
import { fileURLToPath } from 'url';

const __dirname = path.dirname(fileURLToPath(import.meta.url));

describe('generator-node-service:app', function() {
  this.timeout(30000);

  describe('with default options', function() {
    before(async function() {
      await helpers.run(path.join(__dirname, '../generators/app'))
        .withPrompts({
          name: 'my-service',
          description: 'A test service',
          author: 'Test Team',
          language: 'typescript',
          includeDocker: true,
          includeCI: true
        });
    });

    it('creates expected files', function() {
      assert.file([
        'package.json',
        'tsconfig.json',
        'src/index.ts',
        'tests/index.test.ts',
        'Dockerfile',
        '.github/workflows/ci.yml',
        '.gitignore',
        'README.md'
      ]);
    });

    it('substitutes project name in package.json', function() {
      assert.jsonFileContent('package.json', {
        name: 'my-service',
        description: 'A test service'
      });
    });

    it('substitutes project name in README.md', function() {
      assert.fileContent('README.md', 'my-service');
      assert.noFileContent('README.md', '<%= name %>');
      assert.noFileContent('README.md', '<% name %>');
    });

    it('sets correct TypeScript target', function() {
      assert.jsonFileContent('tsconfig.json', {
        compilerOptions: {
          target: 'ES2022',
          strict: true
        }
      });
    });

    it('CI workflow has correct triggers', function() {
      assert.fileContent('.github/workflows/ci.yml', 'on:');
      assert.fileContent('.github/workflows/ci.yml', 'push:');
      assert.fileContent('.github/workflows/ci.yml', 'pull_request:');
    });

    it('Dockerfile uses multi-stage build', function() {
      assert.fileContent('Dockerfile', 'FROM node');
      assert.fileContent('Dockerfile', 'AS builder');
      assert.fileContent('Dockerfile', 'AS production');
    });
  });

  describe('without Docker', function() {
    before(async function() {
      await helpers.run(path.join(__dirname, '../generators/app'))
        .withPrompts({
          name: 'no-docker-service',
          includeDocker: false
        });
    });

    it('does not create Dockerfile', function() {
      assert.noFile('Dockerfile');
      assert.noFile('.dockerignore');
    });

    it('still creates basic service files', function() {
      assert.file(['package.json', 'src/index.ts', '.gitignore']);
    });
  });

  describe('validation', function() {
    it('rejects invalid project names', async function() {
      let errorThrown = false;
      
      try {
        await helpers.run(path.join(__dirname, '../generators/app'))
          .withPrompts({ name: 'Invalid Name With Spaces!' });
      } catch (err) {
        errorThrown = true;
        assert.ok(err.message.includes('invalid') || err.message.includes('name'));
      }
      
      assert.ok(errorThrown, 'Should have thrown for invalid project name');
    });
  });
});

Testing Generator Composability

// test/subgenerator.test.js
describe('generator-node-service:ci', function() {
  describe('GitHub Actions sub-generator', function() {
    before(async function() {
      await helpers.run(path.join(__dirname, '../generators/ci'))
        .withOptions({
          projectName: 'test-project',
          nodeVersion: '20',
          runTests: true,
          runLint: true
        });
    });

    it('creates GitHub Actions workflow', function() {
      assert.file('.github/workflows/ci.yml');
    });

    it('workflow uses correct Node version', function() {
      assert.fileContent('.github/workflows/ci.yml', "node-version: '20'");
    });

    it('workflow runs tests when runTests is true', function() {
      assert.fileContent('.github/workflows/ci.yml', 'npm test');
    });

    it('workflow runs lint when runLint is true', function() {
      assert.fileContent('.github/workflows/ci.yml', 'npm run lint');
    });
  });
});

CI Integration for Template Tests

# .github/workflows/template-tests.yml
name: Template Tests

on:
  push:
    paths:
      - 'backstage/templates/**'
      - 'cookiecutter-*/**'
      - 'generators/**'
      - 'tests/templates/**'

jobs:
  backstage-templates:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pytest pyyaml jsonschema jinja2
      - run: pytest tests/backstage/ -v

  cookiecutter-templates:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install cookiecutter pytest
      - run: pytest tests/cookiecutter/ -v
        timeout-minutes: 10

  yeoman-generators:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"
      - run: npm install
      - run: npm test
        timeout-minutes: 10

Common Template Testing Mistakes

Testing only happy paths — templates receive varied input. Test edge cases: names with hyphens, long names, names that could be SQL-injected, international characters. Many template bugs only appear with unusual inputs.

Not testing the generated tests — if your template includes a test suite, that test suite must pass on the generated output. A template that generates failing tests will erode developer trust in the scaffolding system.

Skipping Docker builds — Docker build failures in generated templates are common and only visible when you actually build the image. Run docker build in CI as part of template testing.

Hardcoded staging values — templates often get copied from real services where staging URLs and database names were hardcoded. Search generated output for internal hostnames, IPs, and credentials.

Not testing template updates — when you update a template, existing projects generated from it won't be updated. But new projects will use the new template. Test both: that the new template generates correctly, and that the update didn't accidentally break compatibility with existing project patterns.

Read more

Testing Atlantis Terraform PR Automation: Workflows, Plan Verification, and Policy Enforcement

Testing Atlantis Terraform PR Automation: Workflows, Plan Verification, and Policy Enforcement

Atlantis automates Terraform plan and apply through pull requests. But Atlantis itself needs testing: workflow configuration, plan output validation, policy enforcement, and server health checks. This guide covers testing Atlantis workflows locally with atlantis-local, validating plan outputs with custom scripts, enforcing Terraform policies with OPA and Conftest, and monitoring Atlantis

By HelpMeTest