Robot Framework

Robot Framework vs pytest: When to Use Each

HelpMeTest

HelpMeTest

— 6 min read
Robot Framework vs pytest: When to Use Each

Robot Framework and pytest both run Python under the hood, both have extensive library ecosystems, and both can test web apps, APIs, and more. Yet they make opposite tradeoffs, attract different users, and serve different needs. Picking the wrong one costs months of accumulated friction.

This is a direct comparison — no hype, no marketing framing.

Syntax Side by Side

The most obvious difference is how tests look.

pytest:

# tests/test_login.py
import pytest
from playwright.sync_api import Page

def test_valid_login_redirects_to_dashboard(page: Page, base_url: str):
    page.goto(f"{base_url}/login")
    page.fill("#username", "admin")
    page.fill("#password", "secret123")
    page.click("button[type='submit']")
    page.wait_for_url("**/dashboard")
    assert page.title() == "Dashboard"

def test_invalid_password_shows_error(page: Page, base_url: str):
    page.goto(f"{base_url}/login")
    page.fill("#username", "admin")
    page.fill("#password", "wrong")
    page.click("button[type='submit']")
    error = page.locator(".error-message")
    assert error.is_visible()
    assert error.text_content() == "Invalid credentials"

Robot Framework:

# tests/login_tests.robot
*** Settings ***
Library    SeleniumLibrary
Resource   ../resources/login_page.resource

*** Test Cases ***
Valid Login Redirects To Dashboard
    Navigate To Login Page
    Enter Credentials    admin    secret123
    Submit Login Form
    Current URL Should Contain    /dashboard
    Page Title Should Be    Dashboard

Invalid Password Shows Error Message
    Navigate To Login Page
    Enter Credentials    admin    wrong
    Submit Login Form
    Element Should Be Visible    css=.error-message
    Element Text Should Be    css=.error-message    Invalid credentials

The RF version reads more like English. The pytest version reads more like code. This isn't a trivial difference — it determines who can maintain your tests.

Keyword-Driven vs Code-Driven

This is the fundamental model difference, not just a syntax preference.

Robot Framework's keyword model:

  • Tests are composed of keywords
  • Keywords abstract implementation details
  • Tests can be written/read by non-developers
  • Logic lives in keyword implementations (Python or RF keywords)
  • Tests describe what happens, not how

pytest's code-driven model:

  • Tests are Python functions
  • Implementation is directly in the test or in helper functions/fixtures
  • Full Python expressiveness always available
  • Tests describe what happens using Python idioms
  • Developers are first-class citizens

The RF model enforces a separation between "what to test" and "how to test it." That's valuable when the people writing tests aren't the same people implementing them. It's overhead when everyone is a developer.

Where the keyword model breaks down:

Complex conditional logic in RF is painful:

# RF — verbose and hard to follow
*** Keywords ***
Process User Based On Role
    [Arguments]    ${user}
    ${is_admin}=    Evaluate    '${user}[role]' == 'admin'
    Run Keyword If    ${is_admin}    Verify Admin Dashboard Features
    ...    ELSE    Verify Viewer Dashboard Features
    ${has_billing}=    Evaluate    '${user}[plan]' == 'pro'
    Run Keyword If    ${has_billing}    Verify Billing Tab Is Visible
    ...    ELSE    Verify Billing Tab Is Hidden
# pytest — straightforward
def test_dashboard_based_on_role(page, user):
    if user['role'] == 'admin':
        verify_admin_dashboard_features(page)
    else:
        verify_viewer_dashboard_features(page)
    
    if user['plan'] == 'pro':
        assert page.locator('.billing-tab').is_visible()
    else:
        assert not page.locator('.billing-tab').is_visible()

Team Skill Requirements

This is often the deciding factor.

For Robot Framework, you need:

  • Someone who understands the keyword model and can architect resource files
  • Python skills for custom library keywords
  • RF-specific knowledge (variable scoping, execution model, settings sections)
  • Discipline to maintain keyword abstractions as the app evolves

For pytest, you need:

  • Python proficiency (at minimum intermediate level)
  • Understanding of pytest fixtures and conftest.py patterns
  • Knowledge of the chosen testing library (Playwright, Selenium, etc.)

If your QA team includes people who don't code, RF gives them a path to writing tests. If everyone is a developer, pytest keeps things in familiar territory with better tooling support (IDE autocomplete, type checking, debuggers).

The hybrid trap: Teams often think "we'll use RF so non-developers can write tests, and developers will implement the keywords." In practice, the non-developers still need significant training to write maintainable RF tests, and the developers end up writing most tests anyway. Be honest about your team composition before committing to RF for this reason.

Ecosystem and Libraries

pytest ecosystem:

  • pytest-playwright — Playwright integration (better browser automation than Selenium)
  • pytest-asyncio — async test support
  • pytest-xdist — parallel execution
  • pytest-cov — coverage reporting
  • responses, httpretty — HTTP mocking
  • factory_boy, faker — test data generation
  • pytest-benchmark — performance testing
  • Hundreds more on PyPI

Robot Framework ecosystem:

  • SeleniumLibrary — Selenium WebDriver
  • robotframework-browser — Playwright-based (newer, better)
  • RequestsLibrary — HTTP/API testing
  • DatabaseLibrary — direct DB access
  • SSHLibrary — SSH automation
  • AppiumLibrary — mobile testing
  • robotframework-pabot — parallel execution

Both ecosystems are mature. pytest's ecosystem is larger and more actively developed — it benefits from the broader Python testing community. RF's ecosystem is more specialized toward acceptance testing use cases.

Library quality: Third-party RF libraries vary significantly in quality and maintenance status. Before adopting a library, check when it was last updated and how many open issues it has. Some popular libraries have gone unmaintained.

Performance

Raw execution speed is comparable — both run Python ultimately. The overhead differences are:

RF overhead per test:

  • Keyword dispatch machinery (~negligible)
  • Logging infrastructure (RF logs everything by default, including keyword arguments)
  • Report generation at the end of execution

pytest overhead per test:

  • Fixture setup/teardown
  • Plugin hooks

In practice, the bottleneck is always your test actions (browser operations, API calls, database queries), not the framework. The framework overhead is noise.

Parallel execution: pytest-xdist vs pabot. Both work. pytest-xdist is simpler to configure for pure Python tests. pabot handles RF-specific concerns (shared test data, resource locks) better.

Reporting

RF reporting — built-in, excellent:

  • log.html: keyword-level trace with timestamps, always generated
  • report.html: high-level summary with tag breakdowns
  • No configuration needed
  • Integrates with Allure for richer reports

pytest reporting — requires plugins:

  • Default output: terminal only
  • pytest-html for HTML reports
  • Allure integration via allure-pytest
  • JUnit XML via --junitxml
  • More flexible but requires setup

If you need rich reports that non-developers can interpret, RF's built-in reports have an edge. If you control the CI environment and are comfortable with plugins, pytest's ecosystem covers everything.

Maintenance Burden

This is where many teams underestimate RF.

RF maintenance costs:

  • Keyword refactoring requires updating all callers (no IDE rename support across .robot files)
  • Locator changes in page resources must be found/replaced across files
  • RF syntax errors appear at runtime, not during editing
  • Limited IDE support (VS Code extension exists but is less capable than Python tooling)
  • Debugging requires reading RF-format tracebacks

pytest maintenance costs:

  • Python refactoring tools work fully (PyCharm, VS Code)
  • Type hints + mypy catch errors before runtime
  • Standard Python debugging (pdb, IDE debugger) works
  • Test isolation via fixtures is explicit and powerful

Over time, pytest projects benefit from Python's mature tooling. RF projects accumulate friction as the codebase grows, especially around finding and updating locators and keywords.

Hybrid Approaches

You don't have to choose one exclusively.

RF for acceptance tests, pytest for unit/integration tests: Common in enterprises. RF handles the high-level acceptance test suite (run by QA), pytest handles unit and integration tests (run by developers). Different audiences, different tools.

tests/
  acceptance/       # Robot Framework
    login.robot
    checkout.robot
  unit/            # pytest
    test_pricing.py
    test_auth.py
  integration/     # pytest
    test_api.py

pytest calling RF tests: You can invoke Robot Framework from pytest using subprocess for mixed reporting:

# tests/test_rf_suite.py
import subprocess
import pytest

def test_robot_acceptance_suite():
    result = subprocess.run(
        ['robot', '--outputdir', '/tmp/rf-results/', 'tests/acceptance/'],
        capture_output=True
    )
    assert result.returncode == 0, f"RF tests failed:\n{result.stdout.decode()}"

RF with Python libraries for complex logic: Write the complex logic in Python library keywords, keep the test DSL in RF:

# libraries/DataValidator.py
from robot.api.deco import keyword

class DataValidator:
    @keyword('JSON Response Should Match Schema')
    def json_should_match_schema(self, response_json, schema_file):
        import jsonschema, json
        with open(schema_file) as f:
            schema = json.load(f)
        jsonschema.validate(response_json, schema)  # Raises on validation failure
*** Test Cases ***
API Response Matches Schema
    ${response}=    GET On Session    api    /users
    JSON Response Should Match Schema    ${response.json()}    schemas/users-list.json

Decision Framework

Answer these questions in order:

1. Who writes and maintains the tests?

  • Mixed team (developers + QA + business analysts): RF is worth considering
  • All developers: pytest
  • Mostly developers with occasional non-developer input: pytest with good helper abstractions

2. What are you testing?

  • Pure unit tests: pytest (RF is wrong for this)
  • Pure API tests: either works, pytest is simpler
  • Web UI acceptance tests: both work, RF has readability advantage
  • Mixed (API + UI + DB): both work, RF's unified report is valuable

3. Do you need rich reports without configuration?

  • Yes, for stakeholders who need to interpret results: RF's built-in reports help
  • No, team-only consumption and you control CI: pytest + plugins is fine

4. What's your existing codebase?

  • Python project with existing pytest: add to pytest, don't add RF
  • Non-Python project (Java, .NET, etc.): RF is language-agnostic, fits better
  • Greenfield: both are options, default to pytest unless criteria above point to RF

5. Long-term maintenance priority?

  • Refactoring and evolving tests aggressively: pytest (better tooling)
  • Stable test suite with infrequent changes: RF's overhead is manageable

The honest default: If you're a developer or developer-led team, default to pytest. It gives you Python's full power, better tooling, and less framework-specific knowledge to maintain. Reach for Robot Framework when you have a genuine need for keyword-driven readability — typically because non-developers are writing or reviewing tests, or because you need RF's specific libraries (SSHLibrary, AppiumLibrary in RF style) or its unified reporting across diverse test types.

RF isn't better or worse than pytest — it's optimized for different constraints. Teams that struggle with RF usually picked it for the wrong reasons. Teams that thrive with it usually have the specific team composition and reporting needs that RF's model addresses.

Read more

Start now free