Testomat.io Getting Started Guide: Modern Test Management for Automated Testing

Testomat.io Getting Started Guide: Modern Test Management for Automated Testing

Testomat.io is a modern test management platform built specifically for automated testing teams. Unlike legacy tools designed for manual QA, Testomat.io understands your test code — it imports directly from Playwright, Cypress, Jest, Cucumber, and other frameworks, giving you a living test suite that stays in sync with your codebase.

This guide walks through getting your first project running on Testomat.io, from account setup through running your first automated test report.

What Testomat.io Is (And Isn't)

Before diving in, it's worth clarifying what Testomat.io does:

What it does:

  • Imports test code from popular frameworks and displays it as a structured test suite
  • Aggregates test run results from CI/CD pipelines
  • Provides test analytics, flakiness tracking, and failure analysis
  • Enables manual test cases alongside automated ones
  • Integrates with Jira, GitHub Issues, and other bug trackers

What it isn't:

  • A test execution engine (it doesn't run your tests)
  • A replacement for Playwright, Cypress, or Jest
  • A visual testing tool or monitoring platform

Testomat.io sits on top of your existing test infrastructure, giving it visibility and management capabilities.

Account Setup and First Project

Creating Your Account

Head to testomat.io and sign up. The free tier gives you one project with up to 1,000 test cases — enough to evaluate the platform seriously.

Once logged in, create a new project:

  1. Click New Project
  2. Name it (e.g., "My App E2E Tests")
  3. Select your primary framework (Playwright, Cypress, WebdriverIO, etc.)
  4. Note your Project ID and API Key from the project settings — you'll need these for CI integration

Project Structure

Testomat.io organizes tests in a hierarchical structure:

Project
└── Suite (top-level grouping)
    └── Suite (nested grouping)
        └── Test

This maps directly to how most test frameworks organize files and describe blocks.

Importing Tests from Your Codebase

This is where Testomat.io shines. Instead of manually creating test cases, you import them from your existing test files.

Installing the CLI

npm install -g @testomatio/check-tests

Or add it as a dev dependency:

npm install --save-dev @testomatio/check-tests

Importing Playwright Tests

If your project uses Playwright with TypeScript:

TESTOMATIO={your-api-key} npx check-tests Playwright "**/*.spec.ts"

This scans your spec files and creates test cases in Testomat.io matching your test structure. Running it again updates the suite — new tests are added, removed tests are flagged as outdated.

Example Playwright test file:

// tests/auth/login.spec.ts
import { test, expect } from '@playwright/test';

test.describe('Login Flow', () => {
  test('logs in with valid credentials', async ({ page }) => {
    await page.goto('/login');
    await page.fill('[data-testid=email]', 'user@example.com');
    await page.fill('[data-testid=password]', 'password123');
    await page.click('[data-testid=submit]');
    await expect(page).toHaveURL('/dashboard');
  });

  test('shows error for invalid credentials', async ({ page }) => {
    await page.goto('/login');
    await page.fill('[data-testid=email]', 'user@example.com');
    await page.fill('[data-testid=password]', 'wrongpassword');
    await page.click('[data-testid=submit]');
    await expect(page.locator('.error-message')).toBeVisible();
  });
});

After importing, Testomat.io shows:

Suite: Login Flow
  ✓ logs in with valid credentials
  ✓ shows error for invalid credentials

Importing Cypress Tests

TESTOMATIO={your-api-key} npx check-tests cypress "**/*.cy.js"

Importing Jest Tests

TESTOMATIO={your-api-key} npx check-tests jest "**/*.test.js"

Importing Cucumber/BDD Tests

Testomat.io handles .feature files directly:

TESTOMATIO={your-api-key} npx check-tests cucumber "**/*.feature"

Feature files are imported with their Given/When/Then structure preserved, making them readable for non-technical stakeholders.

Reporting Test Results to Testomat.io

Importing tests creates the structure. Reporting results fills it with run data.

Installing the Reporter

npm install --save-dev @testomatio/reporter

Playwright Configuration

In your playwright.config.ts:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [
    ['list'],
    ['@testomatio/reporter/lib/adapter/playwright.js', {
      apiKey: process.env.TESTOMATIO,
    }]
  ],
});

Cypress Configuration

In cypress.config.js:

const { defineConfig } = require('cypress');
const testomatioReporter = require('@testomatio/reporter/lib/adapter/cypress');

module.exports = defineConfig({
  e2e: {
    setupNodeEvents(on, config) {
      testomatioReporter(on, config, {
        apiKey: process.env.TESTOMATIO,
      });
      return config;
    },
  },
});

Jest Configuration

In jest.config.js:

module.exports = {
  reporters: [
    'default',
    ['@testomatio/reporter/lib/adapter/jest', {
      apiKey: process.env.TESTOMATIO,
    }]
  ],
};

Running Tests with Reporting

With the reporter configured, set your API key and run tests normally:

TESTOMATIO={your-api-key} npx playwright test

Or using environment variables in .env:

TESTOMATIO=your-api-key-here
npx dotenv -e .env -- npx playwright test

After the run completes, the Testomat.io dashboard shows the results linked to your test cases.

CI/CD Integration

The real value of Testomat.io comes through CI integration. Every pipeline run creates a test run report with pass/fail status, duration, and failure details.

GitHub Actions

name: E2E Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Install Playwright browsers
        run: npx playwright install --with-deps
      
      - name: Run tests
        env:
          TESTOMATIO: ${{ secrets.TESTOMATIO_API_KEY }}
        run: npx playwright test
      
      - name: Upload Playwright report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: playwright-report
          path: playwright-report/

Store your Testomat.io API key as a GitHub secret named TESTOMATIO_API_KEY.

GitLab CI

e2e-tests:
  image: mcr.microsoft.com/playwright:v1.40.0-jammy
  script:
    - npm ci
    - npx playwright test
  variables:
    TESTOMATIO: $TESTOMATIO_API_KEY
  artifacts:
    when: always
    paths:
      - playwright-report/

Jenkins

pipeline {
    agent any
    
    environment {
        TESTOMATIO = credentials('testomatio-api-key')
    }
    
    stages {
        stage('Test') {
            steps {
                sh 'npm ci'
                sh 'npx playwright install --with-deps'
                sh 'npx playwright test'
            }
        }
    }
}

Understanding the Testomat.io Dashboard

Once runs are reporting, you get access to several views:

Test Runs View

The runs list shows each execution with:

  • Status: pass/fail/partial
  • Duration: total run time
  • Environment: branch, build number, environment tag
  • Stats: pass count, fail count, skipped count

Click any run to see individual test results with failure messages and stack traces.

Test Suite View

The suite view shows your test hierarchy with aggregate statistics:

  • How many times each test has run
  • Current pass/fail rate
  • Flakiness score (tests that intermittently fail)
  • Last failure reason

Flakiness detection is particularly useful — Testomat.io automatically flags tests with inconsistent results, helping you prioritize reliability work.

Analytics

The analytics section shows trends over time:

  • Total tests by status
  • Test execution time trends
  • Most frequently failing tests
  • Newly added/removed tests

Tagging and Organizing Tests

Testomat.io supports tags for cross-cutting organization:

// Playwright test with tags
test('@smoke @auth logs in with valid credentials', async ({ page }) => {
  // ...
});

In Testomat.io, you can filter runs and test results by tag:

  • Run only @smoke tests in a lightweight CI check
  • View all @auth test failures across runs
  • Create reports filtered to specific tag groups

Syncing Test IDs Back to Code

Testomat.io can write unique test IDs back into your test files:

TESTOMATIO={your-api-key} npx check-tests Playwright "**/*.spec.ts" --update-ids

This adds IDs to your test titles:

test('@T1a2b3c4 logs in with valid credentials', async ({ page }) => {
  // ...
});

These IDs create a stable link between your code and Testomat.io, even if you rename the test. It also enables direct linking from CI failure reports to the specific test case in Testomat.io.

Manual Tests Alongside Automated Tests

For features that can't yet be automated, Testomat.io supports manual test cases in the same suite:

  1. Navigate to any suite
  2. Click Add Test → Manual Test
  3. Write test steps in Markdown
  4. During a release, assign the manual tests to testers

Manual and automated tests share the same reporting structure, giving you a complete picture of test coverage.

Troubleshooting Common Issues

Tests Not Appearing After Import

  • Verify your API key is correct
  • Check that the glob pattern matches your test files
  • Run with --verbose flag for detailed output: npx check-tests Playwright "**/*.spec.ts" --verbose

Results Not Reporting

  • Confirm the reporter is installed: npm list @testomatio/reporter
  • Verify TESTOMATIO environment variable is set before running tests
  • Check network connectivity to testomat.io from your CI environment

Duplicate Tests After Re-Import

Testomat.io uses test titles to match existing tests. If you renamed tests, it creates new entries rather than updating old ones. Run with --no-detach to prevent removing unmatched tests while you reconcile.

Next Steps

With Testomat.io set up, you have a foundation for serious test management:

  • Connect Jira: Link test failures directly to bug reports
  • Set up environments: Track results separately for staging and production
  • Configure notifications: Get Slack alerts when test suites fail
  • Enable test plans: Create release checklists combining manual and automated tests

Testomat.io works best when it's integrated early in a project's lifecycle — the longer it collects run data, the more useful its flakiness detection and trend analysis become.

For teams running Playwright or Cypress at scale, the reporting and analytics capabilities alone justify the setup overhead. The import-from-code approach means your test management stays in sync with your actual test suite, solving the drift problem that plagues teams using traditional test management tools.

Read more

Start now free