Hermione (Testplane): The Browser Testing Framework from Yandex

Hermione (Testplane): The Browser Testing Framework from Yandex

Hermione is a browser testing framework built by Yandex for running E2E and visual regression tests across multiple browsers in parallel. In 2023, Yandex rebranded it as Testplane. The framework is in production use at Yandex, Mail.ru, and other large Russian tech companies, and it's a practical alternative to Playwright or WebdriverIO for teams that need heavy parallelization and built-in visual testing.

What Testplane Offers

  • Parallel browser execution — run tests across multiple browsers simultaneously, not sequentially
  • Mocha-based test syntax — familiar describe/it with async/await
  • WebdriverIO commands — tests use WebdriverIO's API for browser interaction
  • Built-in retry logic — automatic test retries on failure with configurable count
  • Screenshot-based visual testing — compare screenshots to baseline with configurable tolerance
  • Browser session reuse — option to reuse browser sessions across tests for speed
  • Plugin system — extensible via plugins for reporting, visual comparison, and more

Installation

npm install --save-dev testplane
# or with yarn
yarn add --dev testplane

You need a browser driver. For Chrome:

npm install --save-dev chromedriver

Or use a Selenium Grid / Selenoid setup for multi-browser execution.

Basic Configuration

Create .testplane.conf.js in your project root:

module.exports = {
    baseUrl: 'https://example.com',
    
    gridUrl: 'http://localhost:4444/wd/hub',
    
    browsers: {
        chrome: {
            desiredCapabilities: {
                browserName: 'chrome',
                chromeOptions: {
                    args: ['--headless', '--no-sandbox', '--disable-dev-shm-usage']
                }
            }
        },
        firefox: {
            desiredCapabilities: {
                browserName: 'firefox',
                'moz:firefoxOptions': {
                    args: ['-headless']
                }
            }
        }
    },
    
    sets: {
        desktop: {
            files: ['tests/desktop/**/*.test.js'],
            browsers: ['chrome', 'firefox']
        },
        mobile: {
            files: ['tests/mobile/**/*.test.js'],
            browsers: ['chrome']
        }
    },
    
    // Retry failed tests up to 2 times
    retry: 2,
    
    // Timeout per test in milliseconds
    httpTimeout: 60000,
    testTimeout: 90000,
    
    // Number of parallel sessions per browser
    sessionsPerBrowser: 5,
    
    // Max total parallel sessions
    parallelLimit: 10
};

Writing Tests

Testplane tests use Mocha's describe/it structure with a browser object injected automatically:

describe('Product search', () => {
    it('should display results for a valid query', async ({ browser }) => {
        await browser.url('/search?q=headphones');
        
        const searchResults = await browser.$('.search-results');
        await searchResults.waitForDisplayed({ timeout: 5000 });
        
        const resultCount = await browser.$$('.product-card');
        assert.ok(resultCount.length > 0, 'Should have at least one result');
        
        const firstResult = await browser.$('.product-card:first-child .product-name');
        const productName = await firstResult.getText();
        assert.ok(productName.length > 0, 'Product name should not be empty');
    });

    it('should show no-results message for gibberish query', async ({ browser }) => {
        await browser.url('/search?q=xyzzy123gibberish');
        
        const noResults = await browser.$('.no-results-message');
        await noResults.waitForDisplayed({ timeout: 3000 });
        
        const message = await noResults.getText();
        assert.ok(
            message.includes('No results'),
            `Expected no-results message, got: ${message}`
        );
    });
});

The browser object is WebdriverIO's browser API — commands like $(), $$(), waitForDisplayed(), getText(), setValue(), and click() all work as expected.

Setup and Teardown

Use before, beforeEach, after, and afterEach hooks:

describe('Authenticated features', () => {
    before(async ({ browser }) => {
        // Login once for all tests in this suite
        await browser.url('/login');
        await browser.$('#email').setValue('user@example.com');
        await browser.$('#password').setValue('pass123');
        await browser.$('#login-btn').click();
        await browser.$('.dashboard').waitForDisplayed({ timeout: 5000 });
    });

    it('should show user profile', async ({ browser }) => {
        await browser.url('/profile');
        const name = await browser.$('.profile-name').getText();
        assert.ok(name.length > 0);
    });

    it('should show user orders', async ({ browser }) => {
        await browser.url('/orders');
        const orderList = await browser.$('.order-list');
        await orderList.waitForDisplayed();
        // assertions...
    });
});

Visual Regression Testing

Testplane's killer feature is built-in screenshot comparison. Install the screenshot plugin:

npm install --save-dev testplane-screenshots

Configure it:

module.exports = {
    // ... other config ...
    
    plugins: {
        'testplane-screenshots': {
            enabled: true,
            screenshotsDir: './screenshots',
            diffColor: '#ff0000',
            tolerance: 2.3,  // acceptable pixel difference percentage
            antialiasingTolerance: 4
        }
    }
};

In tests:

it('product card should match baseline', async ({ browser }) => {
    await browser.url('/products/42');
    
    const productCard = await browser.$('.product-detail-card');
    await productCard.waitForDisplayed();
    
    // Take screenshot and compare to baseline
    await browser.assertView('product-detail', '.product-detail-card', {
        ignoreElements: ['.last-updated-timestamp']  // ignore dynamic content
    });
});

On first run, screenshots are saved as baselines. Subsequent runs compare against baselines and fail if pixel differences exceed the tolerance threshold.

Update baselines after intentional UI changes:

npx testplane --update-refs

Parallel Execution

Testplane parallelizes across browsers and across test files. Configure concurrency:

module.exports = {
    // ...
    sessionsPerBrowser: 3,    // 3 parallel Chrome sessions
    parallelLimit: 10,        // max 10 sessions total
    
    browsers: {
        chrome: { /* ... */ },
        firefox: { /* ... */ }
    }
};

With sessionsPerBrowser: 3 and two browsers, Testplane runs up to 6 tests simultaneously. This scales well on CI machines with multiple CPUs.

Selenoid Integration

For multi-browser testing without installing browsers locally, use Selenoid:

# docker-compose.yml
version: '3'
services:
  selenoid:
    image: aerokube/selenoid:latest
    ports:
      - "4444:4444"
    volumes:
      - ./browsers.json:/etc/selenoid/browsers.json
      - /var/run/docker.sock:/var/run/docker.sock

Configure Testplane to point at Selenoid:

module.exports = {
    gridUrl: 'http://localhost:4444/wd/hub',
    
    browsers: {
        chrome: {
            desiredCapabilities: {
                browserName: 'chrome',
                version: '120.0',
                enableVNC: true,  // for Selenoid UI
                enableVideo: false
            }
        },
        firefox: {
            desiredCapabilities: {
                browserName: 'firefox',
                version: '120.0'
            }
        }
    }
};

Running Tests

# Run all tests
npx testplane

# Run specific test file
npx testplane tests/product-search.test.js

# Run in specific browser only
npx testplane --browser chrome

# Run specific set
npx testplane --set desktop

# Update visual baselines
npx testplane --update-refs

# GUI mode (interactive test runner)
npx testplane gui

The GUI mode opens a browser interface where you can run individual tests, see results, and compare visual diffs interactively.

CI Integration

# GitHub Actions
name: Testplane Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm ci

      - name: Start Selenoid
        run: |
          docker-compose up -d selenoid
          # Wait for Selenoid to be ready
          npx wait-on http://localhost:4444/status

      - name: Run Testplane tests
        run: npx testplane

      - name: Upload screenshots on failure
        if: failure()
        uses: actions/upload-artifact@v3
        with:
          name: testplane-screenshots
          path: screenshots/

Testplane vs Playwright

Both frameworks run browser tests. Key differences:

Testplane (Hermione):

  • Uses WebdriverIO API (Selenium-based)
  • Better Selenoid integration for multi-browser grids
  • Built-in visual regression with screenshot comparison
  • Mature retry and parallelization configuration
  • More common in Russian-speaking tech ecosystems

Playwright:

  • Uses Playwright's own browser automation (faster)
  • Built-in multi-browser support without Selenium Grid
  • Better cross-browser isolation out of the box
  • Larger English-language community and documentation

For projects already using Selenium Grid or Selenoid infrastructure, Testplane integrates well. For new projects without existing Selenium infrastructure, Playwright typically has lower setup overhead.

Production Monitoring

Testplane tests run during CI — they verify behavior at deployment time. For production confidence between deployments, HelpMeTest monitors your critical user flows continuously. No WebDriver, no Selenoid, no Node.js build pipeline required — just plain-English test descriptions running on a schedule.

Summary

Testplane (formerly Hermione) is a production-grade browser testing framework optimized for parallelization and visual regression. If you're running a large suite across multiple browsers and need built-in screenshot comparison, it's worth considering alongside Playwright and WebdriverIO. The WebdriverIO command API means existing Selenium knowledge transfers directly, and the Selenoid integration is first-class.

Read more

Start now free