Platform Engineering

Developer Portal Testing: How to QA Backstage, API Catalogs, and Self-Service UIs

HelpMeTest

HelpMeTest

— 10 min read
Developer Portal Testing: How to QA Backstage, API Catalogs, and Self-Service UIs

Developer portals are the front door to your internal developer platform. When the portal is broken — catalog entries missing, API docs stale, scaffolder failing silently — developers lose trust in the platform. This guide covers how to test developer portals built on Backstage (and similar tools): catalog data integrity, API documentation accuracy, self-service workflow E2E testing, search functionality, and RBAC enforcement.

Key Takeaways

Test the catalog's accuracy, not just its availability. A Backstage catalog that loads fine but shows stale owners, wrong lifecycle states, or missing components is a broken catalog. Test that catalog entries reflect ground truth.

API catalog testing is documentation testing. Every API in your catalog should be tested for schema accuracy (does the live API match the OpenAPI spec?), endpoint reachability, and documented examples that actually work.

Self-service forms need both happy-path and validation tests. Backstage scaffolder templates commonly break on edge-case inputs (long names, special characters, existing resources). Test the validation rules, not just the success path.

Test RBAC at the portal level, not just the backend. Backstage's permission framework controls what users see and can do in the portal UI. Test that unauthorized users see appropriate empty states, not errors or accidentally exposed data.

Search is the portal's most-used feature. A developer who can't find the service they need via search will file a ticket instead. Test search relevance, not just search response.

Why Developer Portal Testing Is Neglected

Developer portals get treated as internal tools with lower quality bars than customer-facing products. This is a mistake. The developer portal serves every engineer in the organization. When it fails:

  • Developers can't find who owns a service → wrong team gets paged
  • API catalog shows outdated spec → integration bugs in production
  • Scaffolder fails silently → 30 minutes wasted by developer, platform ticket filed
  • RBAC is too loose → developers see resources they shouldn't
  • Search returns irrelevant results → developers work around the portal entirely

Each of these is a trust-destroying failure. Recovering developer trust in a portal is much harder than preventing the failure.

Test Layer 1: Catalog Data Integrity

The catalog is only valuable if its data is accurate. Test it continuously, not just on deploy.

Catalog Freshness Test

Every catalog entity should have been synced recently. Stale entities indicate a broken discovery plugin or a deleted service that was never removed.

// tests/catalog/freshness.spec.ts
import { test, expect } from '@playwright/test';

test('catalog entities were synced within the last hour', async ({ request }) => {
  const response = await request.get(
    'https://portal.internal/api/catalog/entities?limit=100',
    { headers: { Authorization: `Bearer ${process.env.BACKSTAGE_TOKEN}` } }
  );
  expect(response.status()).toBe(200);
  
  const entities = await response.json();
  const oneHourAgo = new Date(Date.now() - 60 * 60 * 1000);
  
  const staleEntities = entities.filter((entity: any) => {
    const lastSynced = new Date(entity.metadata?.annotations?.['backstage.io/source-last-modified'] || 0);
    return lastSynced < oneHourAgo;
  });

  if (staleEntities.length > 0) {
    console.log('Stale entities:', staleEntities.map((e: any) => e.metadata.name));
  }
  
  // Allow up to 5% stale (some entities may have slow discovery cycles)
  const stalePercent = (staleEntities.length / entities.length) * 100;
  expect(stalePercent).toBeLessThan(5);
});

Catalog Accuracy Test: Owners Match Real Teams

test('catalog component owners are valid teams', async ({ request }) => {
  // Fetch all teams from GitHub (or your team source of truth)
  const teamsResponse = await request.get(
    `https://api.github.com/orgs/${process.env.GITHUB_ORG}/teams`,
    { headers: { Authorization: `Bearer ${process.env.GITHUB_TOKEN}` } }
  );
  const validTeams = (await teamsResponse.json()).map((t: any) => t.slug);

  // Fetch all catalog components
  const catalogResponse = await request.get(
    'https://portal.internal/api/catalog/entities?filter=kind=Component',
    { headers: { Authorization: `Bearer ${process.env.BACKSTAGE_TOKEN}` } }
  );
  const components = await catalogResponse.json();

  const orphanedComponents = components.filter((c: any) => {
    const owner = c.spec?.owner?.replace('group:', '').replace('user:', '');
    return owner && !validTeams.includes(owner);
  });

  if (orphanedComponents.length > 0) {
    console.error('Components with invalid owners:', 
      orphanedComponents.map((c: any) => `${c.metadata.name} (owner: ${c.spec.owner})`));
  }

  expect(orphanedComponents.length).toBe(0);
});

Catalog Coverage Test: Every Service Has a Catalog Entry

#!/bin/bash
<span class="hljs-comment"># tests/catalog/coverage.sh
<span class="hljs-comment"># Every service in GitHub should have a Backstage catalog entry

GITHUB_SERVICES=$(gh repo list <span class="hljs-string">"$GITHUB_ORG" --topic <span class="hljs-string">"microservice" --json name -q <span class="hljs-string">'.[].name')
CATALOG_SERVICES=$(curl -s -H <span class="hljs-string">"Authorization: Bearer $BACKSTAGE_TOKEN" \
  <span class="hljs-string">"https://portal.internal/api/catalog/entities?filter=kind=Component" \
  <span class="hljs-pipe">| jq -r <span class="hljs-string">'.[].metadata.name')

MISSING=0
<span class="hljs-keyword">for svc <span class="hljs-keyword">in <span class="hljs-variable">$GITHUB_SERVICES; <span class="hljs-keyword">do
  <span class="hljs-keyword">if ! <span class="hljs-built_in">echo <span class="hljs-string">"$CATALOG_SERVICES" <span class="hljs-pipe">| grep -q <span class="hljs-string">"^$svc$"; <span class="hljs-keyword">then
    <span class="hljs-built_in">echo <span class="hljs-string">"MISSING from catalog: $svc"
    MISSING=$((MISSING + <span class="hljs-number">1))
  <span class="hljs-keyword">fi
<span class="hljs-keyword">done

<span class="hljs-built_in">echo <span class="hljs-string">"Total services: $(echo "$GITHUB_SERVICES" <span class="hljs-pipe">| wc -l)"
<span class="hljs-built_in">echo <span class="hljs-string">"Missing from catalog: $MISSING"

<span class="hljs-comment"># Fail if more than 10% are missing
TOTAL=$(<span class="hljs-built_in">echo <span class="hljs-string">"$GITHUB_SERVICES" <span class="hljs-pipe">| <span class="hljs-built_in">wc -l)
THRESHOLD=$((TOTAL / <span class="hljs-number">10))
[ <span class="hljs-variable">$MISSING -le <span class="hljs-variable">$THRESHOLD ] <span class="hljs-pipe">|| <span class="hljs-built_in">exit 1

Test Layer 2: API Catalog Testing

API catalogs are only valuable if the APIs they document actually work as documented.

Schema Accuracy Test: Live API Matches OpenAPI Spec

// tests/api-catalog/schema-accuracy.spec.ts
import { test, expect } from '@playwright/test';
import SwaggerParser from '@apidevtools/swagger-parser';

const API_CATALOG_ENTRIES = [
  { name: 'user-service', specUrl: 'https://portal.internal/api/catalog/entities/by-name/api/default/user-service' },
  { name: 'order-service', specUrl: 'https://portal.internal/api/catalog/entities/by-name/api/default/order-service' },
];

for (const api of API_CATALOG_ENTRIES) {
  test(`${api.name}: OpenAPI spec is valid`, async ({ request }) => {
    const entityResponse = await request.get(api.specUrl, {
      headers: { Authorization: `Bearer ${process.env.BACKSTAGE_TOKEN}` }
    });
    expect(entityResponse.status()).toBe(200);
    
    const entity = await entityResponse.json();
    const specUrl = entity.spec?.definition?.['$text'];
    
    // Validate the OpenAPI spec is parseable
    const parsed = await SwaggerParser.validate(specUrl);
    expect(parsed.info).toBeDefined();
    expect(parsed.paths).toBeDefined();
  });
}

Endpoint Reachability Test

test('catalog APIs have reachable base URLs', async ({ request }) => {
  const apisResponse = await request.get(
    'https://portal.internal/api/catalog/entities?filter=kind=API',
    { headers: { Authorization: `Bearer ${process.env.BACKSTAGE_TOKEN}` } }
  );
  const apis = await apisResponse.json();

  for (const api of apis) {
    const baseUrl = api.metadata.annotations?.['backstage.io/techdocs-ref'];
    if (!baseUrl) continue;

    const healthResponse = await request.get(`${baseUrl}/health`, {
      timeout: 5000,
    }).catch(() => null);

    if (!healthResponse || healthResponse.status() >= 500) {
      console.warn(`API ${api.metadata.name} health check failed`);
    }
    // Don't fail the test — just report; some APIs may be intentionally internal-only
  }
});

Test Layer 3: Self-Service Workflow Testing

Scaffolder Template Validation Testing

// tests/self-service/scaffolder-validation.spec.ts
import { test, expect } from '@playwright/test';

test.describe('New Microservice template — input validation', () => {
  test.beforeEach(async ({ page }) => {
    await page.goto('https://portal.internal/create/templates/default/new-microservice');
    await page.getByRole('button', { name: /enter.*guest/i }).click().catch(() => {});
  });

  test('rejects service name with spaces', async ({ page }) => {
    await page.getByLabel('Service Name').fill('my service name');
    await page.getByRole('button', { name: /next/i }).click();
    
    await expect(page.getByText(/invalid|not allowed|must be/i)).toBeVisible();
    // Should NOT proceed to step 2
    await expect(page.getByLabel('Owner Team')).not.toBeVisible();
  });

  test('rejects service name that already exists', async ({ page }) => {
    // 'user-service' is an existing service in the catalog
    await page.getByLabel('Service Name').fill('user-service');
    await page.getByRole('button', { name: /next/i }).click();
    
    await expect(page.getByText(/already exists|taken|conflict/i)).toBeVisible();
  });

  test('rejects service name over 63 characters (Kubernetes limit)', async ({ page }) => {
    await page.getByLabel('Service Name').fill('a'.repeat(64));
    await page.getByRole('button', { name: /next/i }).click();
    
    await expect(page.getByText(/too long|max.*63|character limit/i)).toBeVisible();
  });

  test('accepts valid service name and advances to step 2', async ({ page }) => {
    await page.getByLabel('Service Name').fill('valid-service-name');
    await page.getByRole('button', { name: /next/i }).click();
    
    await expect(page.getByLabel('Owner Team')).toBeVisible();
  });
});

Scaffolder Completion E2E Test

test('scaffolder creates service and shows all output links', async ({ page }) => {
  const SERVICE_NAME = `test-${Date.now()}`;
  
  await page.goto('https://portal.internal/create/templates/default/new-microservice');
  await page.getByLabel('Service Name').fill(SERVICE_NAME);
  await page.getByRole('button', { name: /next/i }).click();
  await page.getByLabel('Owner Team').fill('platform-team');
  await page.getByRole('button', { name: /create/i }).click();

  // Wait for completion
  await expect(page.getByText(/finished/i)).toBeVisible({ timeout: 120_000 });

  // All expected links should be present
  await expect(page.getByRole('link', { name: /github/i })).toBeVisible();
  await expect(page.getByRole('link', { name: /catalog/i })).toBeVisible();
  await expect(page.getByRole('link', { name: /argocd|pipeline/i })).toBeVisible();
  
  // No error messages in output
  await expect(page.getByText(/error|failed|exception/i)).not.toBeVisible();
});

Test Layer 4: Search and Navigation

Search Relevance Test

// tests/portal/search.spec.ts
import { test, expect } from '@playwright/test';

const SEARCH_SCENARIOS = [
  { query: 'user service', expectedFirstResult: 'user-service' },
  { query: 'payment api', expectedFirstResult: 'payment-service' },
  { query: 'postgres database', expectedFirstResult: 'postgres-instance' },
];

for (const scenario of SEARCH_SCENARIOS) {
  test(`search for "${scenario.query}" returns "${scenario.expectedFirstResult}" as top result`, async ({ page }) => {
    await page.goto('https://portal.internal/search');
    await page.getByRole('textbox', { name: /search/i }).fill(scenario.query);
    await page.keyboard.press('Enter');

    const firstResult = page.locator('[data-testid="search-result"]').first();
    await expect(firstResult).toBeVisible({ timeout: 10_000 });
    await expect(firstResult).toContainText(scenario.expectedFirstResult);
  });
}

test('search with no results shows helpful empty state', async ({ page }) => {
  await page.goto('https://portal.internal/search');
  await page.getByRole('textbox', { name: /search/i }).fill('xyzzy-nonexistent-service-12345');
  await page.keyboard.press('Enter');

  await expect(page.getByText(/no results|nothing found|try a different/i)).toBeVisible();
  // Should NOT show an error or loading spinner forever
  await expect(page.getByRole('progressbar')).not.toBeVisible();
});
test('entity page links are stable and resolve correctly', async ({ page }) => {
  // Direct URL to a known entity
  await page.goto('https://portal.internal/catalog/default/component/user-service');
  
  await expect(page.getByRole('heading', { name: 'user-service' })).toBeVisible({ timeout: 10_000 });
  await expect(page.getByText(/owner/i)).toBeVisible();
  
  // Tabs should load without errors
  for (const tab of ['Overview', 'CI/CD', 'Docs', 'Dependencies']) {
    const tabButton = page.getByRole('tab', { name: tab });
    if (await tabButton.isVisible()) {
      await tabButton.click();
      await expect(page.getByRole('progressbar')).not.toBeVisible({ timeout: 10_000 });
    }
  }
});

Test Layer 5: RBAC and Permission Testing

Unauthorized User Sees Empty States, Not Errors

// tests/portal/rbac.spec.ts
import { test, expect } from '@playwright/test';

test.describe('RBAC: restricted resources', () => {
  test.use({ storageState: 'tests/.auth/limited-user.json' });

  test('limited user cannot see admin-only settings', async ({ page }) => {
    await page.goto('https://portal.internal/settings/admin');
    
    // Should redirect or show access denied — never a 500 or raw error
    const isAccessDenied = await page.getByText(/access denied|permission|not authorized/i).isVisible();
    const isRedirected = page.url().includes('/catalog') || page.url().includes('/home');
    
    expect(isAccessDenied || isRedirected).toBe(true);
  });

  test('limited user sees empty catalog for restricted teams', async ({ request }) => {
    // The limited user should not see components owned by "secrets-team"
    const response = await request.get(
      'https://portal.internal/api/catalog/entities?filter=kind=Component,spec.owner=group:secrets-team',
      { headers: { Authorization: `Bearer ${process.env.LIMITED_USER_TOKEN}` } }
    );
    
    const entities = await response.json();
    expect(entities).toHaveLength(0);
  });
});

Create Auth State for RBAC Tests

// tests/setup/create-limited-user-auth.ts
import { chromium } from '@playwright/test';

async function createLimitedUserAuth() {
  const browser = await chromium.launch();
  const context = await browser.newContext();
  const page = await context.newPage();
  
  await page.goto('https://portal.internal');
  // Log in as a limited user via your auth provider
  await page.getByLabel('Email').fill(process.env.LIMITED_USER_EMAIL!);
  await page.getByLabel('Password').fill(process.env.LIMITED_USER_PASSWORD!);
  await page.getByRole('button', { name: /sign in/i }).click();
  await page.waitForURL('**/catalog');
  
  await context.storageState({ path: 'tests/.auth/limited-user.json' });
  await browser.close();
}

createLimitedUserAuth();

Test Layer 6: Backstage Catalog API Contract Tests

Test that the Backstage catalog API adheres to its expected contract:

// tests/api/catalog-contract.spec.ts
import { test, expect } from '@playwright/test';

test('catalog entities endpoint returns expected shape', async ({ request }) => {
  const response = await request.get(
    'https://portal.internal/api/catalog/entities?limit=1',
    { headers: { Authorization: `Bearer ${process.env.BACKSTAGE_TOKEN}` } }
  );
  
  expect(response.status()).toBe(200);
  const [entity] = await response.json();
  
  // Required fields per Backstage entity model
  expect(entity).toMatchObject({
    apiVersion: expect.stringMatching(/backstage\.io/),
    kind: expect.any(String),
    metadata: {
      name: expect.any(String),
      namespace: expect.any(String),
      uid: expect.any(String),
    },
  });
});

test('catalog supports pagination', async ({ request }) => {
  const page1 = await request.get(
    'https://portal.internal/api/catalog/entities?limit=5&offset=0',
    { headers: { Authorization: `Bearer ${process.env.BACKSTAGE_TOKEN}` } }
  );
  const page2 = await request.get(
    'https://portal.internal/api/catalog/entities?limit=5&offset=5',
    { headers: { Authorization: `Bearer ${process.env.BACKSTAGE_TOKEN}` } }
  );
  
  const entities1 = await page1.json();
  const entities2 = await page2.json();
  
  expect(entities1).toHaveLength(5);
  // Pages should not overlap
  const names1 = new Set(entities1.map((e: any) => e.metadata.uid));
  const names2 = new Set(entities2.map((e: any) => e.metadata.uid));
  const overlap = [...names1].filter(uid => names2.has(uid));
  expect(overlap).toHaveLength(0);
});

CI Pipeline for Developer Portal Tests

# .github/workflows/portal-tests.yaml
name: Developer Portal Tests

on:
  schedule:
    - cron: '*/15 * * * *'  # Every 15 minutes
  push:
    branches: [main]

jobs:
  catalog-integrity:
    name: Catalog Data Integrity
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20' }
      - run: npm ci
      - run: npx playwright test tests/catalog/
        env:
          BACKSTAGE_TOKEN: ${{ secrets.BACKSTAGE_TOKEN }}
          GITHUB_TOKEN: ${{ secrets.GH_TOKEN }}

  self-service-e2e:
    name: Self-Service Workflows
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20' }
      - run: npm ci && npx playwright install chromium
      - run: npx playwright test tests/self-service/
        env:
          BACKSTAGE_TOKEN: ${{ secrets.BACKSTAGE_TOKEN }}
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: playwright-report
          path: playwright-report/

  rbac:
    name: RBAC Enforcement
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20' }
      - run: npm ci && npx playwright install chromium
      - name: Create limited user auth state
        run: npx ts-node tests/setup/create-limited-user-auth.ts
        env:
          LIMITED_USER_EMAIL: ${{ secrets.LIMITED_USER_EMAIL }}
          LIMITED_USER_PASSWORD: ${{ secrets.LIMITED_USER_PASSWORD }}
      - run: npx playwright test tests/portal/rbac.spec.ts

Developer Portal Testing Checklist

Before releasing a developer portal update:

Catalog:

  • All catalog entity kinds (Component, API, Resource, System, Domain) load correctly
  • Entity freshness < 1 hour for actively-managed entities
  • Owners resolve to valid teams
  • Dependency relationships render correctly in the graph view

API Catalog:

  • OpenAPI specs parse without errors
  • Spec examples are accurate (at least one example per endpoint)
  • API base URLs are reachable

Self-Service:

  • All scaffolder templates pass validation tests
  • Validation rules reject known invalid inputs (special chars, length limits, conflicts)
  • Successful scaffolder run creates all expected outputs
  • Failed scaffolder run shows clear error — no silent failure

Search:

  • Top 10 most-searched entities return expected results
  • Empty search state is helpful, not an error

RBAC:

  • Unauthorized users see access-denied states, not 500 errors
  • Restricted resources are not visible to limited users via API or UI

Developer portals earn trust through reliability, not features. A portal that shows accurate catalog data, works every time a developer runs a scaffolder template, and never exposes the wrong data to the wrong person is worth more than a portal with 50 features and flaky behavior. Build the test suite described here, run it on a cron every 15 minutes, and treat a failing portal test with the same urgency as a failing production service.

HelpMeTest can monitor your developer portal continuously — write the assertions in plain English and get alerted the moment the catalog drifts, a scaffolder breaks, or RBAC enforcement regresses.

Read more