Testomat.io Getting Started Guide: Modern Test Management for Automated Testing
Testomat.io is a modern test management platform built specifically for automated testing teams. Unlike legacy tools designed for manual QA, Testomat.io understands your test code — it imports directly from Playwright, Cypress, Jest, Cucumber, and other frameworks, giving you a living test suite that stays in sync with your codebase.
This guide walks through getting your first project running on Testomat.io, from account setup through running your first automated test report.
What Testomat.io Is (And Isn't)
Before diving in, it's worth clarifying what Testomat.io does:
What it does:
- Imports test code from popular frameworks and displays it as a structured test suite
- Aggregates test run results from CI/CD pipelines
- Provides test analytics, flakiness tracking, and failure analysis
- Enables manual test cases alongside automated ones
- Integrates with Jira, GitHub Issues, and other bug trackers
What it isn't:
- A test execution engine (it doesn't run your tests)
- A replacement for Playwright, Cypress, or Jest
- A visual testing tool or monitoring platform
Testomat.io sits on top of your existing test infrastructure, giving it visibility and management capabilities.
Account Setup and First Project
Creating Your Account
Head to testomat.io and sign up. The free tier gives you one project with up to 1,000 test cases — enough to evaluate the platform seriously.
Once logged in, create a new project:
- Click New Project
- Name it (e.g., "My App E2E Tests")
- Select your primary framework (Playwright, Cypress, WebdriverIO, etc.)
- Note your Project ID and API Key from the project settings — you'll need these for CI integration
Project Structure
Testomat.io organizes tests in a hierarchical structure:
Project
└── Suite (top-level grouping)
└── Suite (nested grouping)
└── TestThis maps directly to how most test frameworks organize files and describe blocks.
Importing Tests from Your Codebase
This is where Testomat.io shines. Instead of manually creating test cases, you import them from your existing test files.
Installing the CLI
npm install -g @testomatio/check-testsOr add it as a dev dependency:
npm install --save-dev @testomatio/check-testsImporting Playwright Tests
If your project uses Playwright with TypeScript:
TESTOMATIO={your-api-key} npx check-tests Playwright "**/*.spec.ts"This scans your spec files and creates test cases in Testomat.io matching your test structure. Running it again updates the suite — new tests are added, removed tests are flagged as outdated.
Example Playwright test file:
// tests/auth/login.spec.ts
import { test, expect } from '@playwright/test';
test.describe('Login Flow', () => {
test('logs in with valid credentials', async ({ page }) => {
await page.goto('/login');
await page.fill('[data-testid=email]', 'user@example.com');
await page.fill('[data-testid=password]', 'password123');
await page.click('[data-testid=submit]');
await expect(page).toHaveURL('/dashboard');
});
test('shows error for invalid credentials', async ({ page }) => {
await page.goto('/login');
await page.fill('[data-testid=email]', 'user@example.com');
await page.fill('[data-testid=password]', 'wrongpassword');
await page.click('[data-testid=submit]');
await expect(page.locator('.error-message')).toBeVisible();
});
});After importing, Testomat.io shows:
Suite: Login Flow
✓ logs in with valid credentials
✓ shows error for invalid credentialsImporting Cypress Tests
TESTOMATIO={your-api-key} npx check-tests cypress "**/*.cy.js"Importing Jest Tests
TESTOMATIO={your-api-key} npx check-tests jest "**/*.test.js"Importing Cucumber/BDD Tests
Testomat.io handles .feature files directly:
TESTOMATIO={your-api-key} npx check-tests cucumber "**/*.feature"Feature files are imported with their Given/When/Then structure preserved, making them readable for non-technical stakeholders.
Reporting Test Results to Testomat.io
Importing tests creates the structure. Reporting results fills it with run data.
Installing the Reporter
npm install --save-dev @testomatio/reporterPlaywright Configuration
In your playwright.config.ts:
import { defineConfig } from '@playwright/test';
export default defineConfig({
reporter: [
['list'],
['@testomatio/reporter/lib/adapter/playwright.js', {
apiKey: process.env.TESTOMATIO,
}]
],
});Cypress Configuration
In cypress.config.js:
const { defineConfig } = require('cypress');
const testomatioReporter = require('@testomatio/reporter/lib/adapter/cypress');
module.exports = defineConfig({
e2e: {
setupNodeEvents(on, config) {
testomatioReporter(on, config, {
apiKey: process.env.TESTOMATIO,
});
return config;
},
},
});Jest Configuration
In jest.config.js:
module.exports = {
reporters: [
'default',
['@testomatio/reporter/lib/adapter/jest', {
apiKey: process.env.TESTOMATIO,
}]
],
};Running Tests with Reporting
With the reporter configured, set your API key and run tests normally:
TESTOMATIO={your-api-key} npx playwright testOr using environment variables in .env:
TESTOMATIO=your-api-key-herenpx dotenv -e .env -- npx playwright testAfter the run completes, the Testomat.io dashboard shows the results linked to your test cases.
CI/CD Integration
The real value of Testomat.io comes through CI integration. Every pipeline run creates a test run report with pass/fail status, duration, and failure details.
GitHub Actions
name: E2E Tests
on:
push:
branches: [main]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Install dependencies
run: npm ci
- name: Install Playwright browsers
run: npx playwright install --with-deps
- name: Run tests
env:
TESTOMATIO: ${{ secrets.TESTOMATIO_API_KEY }}
run: npx playwright test
- name: Upload Playwright report
uses: actions/upload-artifact@v4
if: always()
with:
name: playwright-report
path: playwright-report/Store your Testomat.io API key as a GitHub secret named TESTOMATIO_API_KEY.
GitLab CI
e2e-tests:
image: mcr.microsoft.com/playwright:v1.40.0-jammy
script:
- npm ci
- npx playwright test
variables:
TESTOMATIO: $TESTOMATIO_API_KEY
artifacts:
when: always
paths:
- playwright-report/Jenkins
pipeline {
agent any
environment {
TESTOMATIO = credentials('testomatio-api-key')
}
stages {
stage('Test') {
steps {
sh 'npm ci'
sh 'npx playwright install --with-deps'
sh 'npx playwright test'
}
}
}
}Understanding the Testomat.io Dashboard
Once runs are reporting, you get access to several views:
Test Runs View
The runs list shows each execution with:
- Status: pass/fail/partial
- Duration: total run time
- Environment: branch, build number, environment tag
- Stats: pass count, fail count, skipped count
Click any run to see individual test results with failure messages and stack traces.
Test Suite View
The suite view shows your test hierarchy with aggregate statistics:
- How many times each test has run
- Current pass/fail rate
- Flakiness score (tests that intermittently fail)
- Last failure reason
Flakiness detection is particularly useful — Testomat.io automatically flags tests with inconsistent results, helping you prioritize reliability work.
Analytics
The analytics section shows trends over time:
- Total tests by status
- Test execution time trends
- Most frequently failing tests
- Newly added/removed tests
Tagging and Organizing Tests
Testomat.io supports tags for cross-cutting organization:
// Playwright test with tags
test('@smoke @auth logs in with valid credentials', async ({ page }) => {
// ...
});In Testomat.io, you can filter runs and test results by tag:
- Run only
@smoketests in a lightweight CI check - View all
@authtest failures across runs - Create reports filtered to specific tag groups
Syncing Test IDs Back to Code
Testomat.io can write unique test IDs back into your test files:
TESTOMATIO={your-api-key} npx check-tests Playwright "**/*.spec.ts" --update-idsThis adds IDs to your test titles:
test('@T1a2b3c4 logs in with valid credentials', async ({ page }) => {
// ...
});These IDs create a stable link between your code and Testomat.io, even if you rename the test. It also enables direct linking from CI failure reports to the specific test case in Testomat.io.
Manual Tests Alongside Automated Tests
For features that can't yet be automated, Testomat.io supports manual test cases in the same suite:
- Navigate to any suite
- Click Add Test → Manual Test
- Write test steps in Markdown
- During a release, assign the manual tests to testers
Manual and automated tests share the same reporting structure, giving you a complete picture of test coverage.
Troubleshooting Common Issues
Tests Not Appearing After Import
- Verify your API key is correct
- Check that the glob pattern matches your test files
- Run with
--verboseflag for detailed output:npx check-tests Playwright "**/*.spec.ts" --verbose
Results Not Reporting
- Confirm the reporter is installed:
npm list @testomatio/reporter - Verify
TESTOMATIOenvironment variable is set before running tests - Check network connectivity to
testomat.iofrom your CI environment
Duplicate Tests After Re-Import
Testomat.io uses test titles to match existing tests. If you renamed tests, it creates new entries rather than updating old ones. Run with --no-detach to prevent removing unmatched tests while you reconcile.
Next Steps
With Testomat.io set up, you have a foundation for serious test management:
- Connect Jira: Link test failures directly to bug reports
- Set up environments: Track results separately for staging and production
- Configure notifications: Get Slack alerts when test suites fail
- Enable test plans: Create release checklists combining manual and automated tests
Testomat.io works best when it's integrated early in a project's lifecycle — the longer it collects run data, the more useful its flakiness detection and trend analysis become.
For teams running Playwright or Cypress at scale, the reporting and analytics capabilities alone justify the setup overhead. The import-from-code approach means your test management stays in sync with your actual test suite, solving the drift problem that plagues teams using traditional test management tools.