SauceLabs

SauceLabs Insights: Using Test Analytics to Find Flaky Tests and Bottlenecks

17 Sep 2026 — 5 min read

SauceLabs Insights gives you aggregated analytics across all your test runs — flakiness scores, pass rate trends, slowest tests, and failure clustering. This post covers using the Insights dashboard to find and fix flaky tests, analyzing failure patterns, using the SauceLabs API to build custom reports, and integrating Backtrace for error tracking.

What SauceLabs Insights Tracks

SauceLabs Insights (formerly SauceLabs Analytics) collects data from every test session:

Pass/fail history per test, per build, per browser
Flakiness score — tests that sometimes pass and sometimes fail with no code changes
Duration trends — tests getting slower over time
Error clustering — groups tests by failure message to find common root causes
Browser-specific failures — tests that fail on Safari but pass on Chrome

Access it at app.saucelabs.com under the Insights tab.

Flaky Test Detection

A flaky test is one that fails on some runs and passes on others without any code change. SauceLabs calculates a flakiness score (0-100) based on pass/fail alternation history.

What the Flakiness Score Means

Score	Meaning	Action
0-10	Stable	No action needed
11-30	Occasionally flaky	Monitor, investigate if score rises
31-60	Moderately flaky	Fix before next sprint
61-100	Highly flaky	Fix immediately or quarantine

Finding Flaky Tests via API

# Get tests sorted by flakiness score
curl -u "$SAUCE_USERNAME:$SAUCE_ACCESS_KEY" \
  "https://api.us-west-1.saucelabs.com/v2/insights/trends/tests/flaky?limit=20&time_range=7d" \
  | jq '.tests[] | {name: .test_name, score: .flakiness_score, failures: .failure_count}'

Example response:

[
  {"name": "Checkout - payment step on Safari", "score": 78, "failures": 14},
  {"name": "Login - SSO flow on Windows", "score": 45, "failures": 7},
  {"name": "Search - autocomplete on IE11", "score": 31, "failures": 4}
]

Common Causes of Flakiness

Race conditions (most common):

# Flaky: no wait for dynamic content
element = driver.find_element(By.ID, "dynamic-result")
assert element.text == "Success"

# Fixed: explicit wait
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

element = WebDriverWait(driver, 10).until(
    EC.text_to_be_present_in_element((By.ID, "dynamic-result"), "Success")
)

Animation/transition timing:

# Wait for CSS animations to complete
driver.execute_script("""
    return new Promise(resolve => {
        document.getAnimations().forEach(a => a.finish());
        resolve();
    });
""")

Network timing in SauceLabs VMs: Add explicit waits for network requests rather than using time.sleep(). Use wait.until with a condition that checks the actual DOM state.

Test History and Trends

Pass Rate Over Time

# Get daily pass rate for last 30 days
curl -u "$SAUCE_USERNAME:$SAUCE_ACCESS_KEY" \
  "https://api.us-west-1.saucelabs.com/v2/insights/trends/builds?time_range=30d&interval=day" \
  | jq '.data[] | {date: .date, pass_rate: (.passed / .total * 100 | round)}'

Build-Level Metrics

# Get metrics for a specific build
BUILD="v2.4.0"
curl -u "$SAUCE_USERNAME:$SAUCE_ACCESS_KEY" \
  "https://api.us-west-1.saucelabs.com/v2/insights/builds?name=$BUILD" \
  | jq '.builds[] | {
      name: .name,
      total: .total_tests,
      passed: .passed,
      failed: .failed,
      duration_ms: .total_duration
    }'

Slowest Tests

# Find the 10 slowest tests
curl -u "$SAUCE_USERNAME:$SAUCE_ACCESS_KEY" \
  "https://api.us-west-1.saucelabs.com/v2/insights/trends/tests?sort_by=duration&limit=10&time_range=7d" \
  | jq '.tests[] | {name: .test_name, avg_duration_s: (.avg_duration / 1000 | round)}'

Failure Analysis

Error Clustering

SauceLabs groups similar failures by error message. Use this to find systemic issues vs one-off failures.

# Get error clusters for failed tests
curl -u "$SAUCE_USERNAME:$SAUCE_ACCESS_KEY" \
  "https://api.us-west-1.saucelabs.com/v2/insights/errors?time_range=7d&limit=10" \
  | jq '.errors[] | {
      message: .error_message,
      count: .occurrence_count,
      affected_tests: .test_count
    }'

If you see "TimeoutException: Waiting for element" appearing 40 times across 15 tests, that points to a performance regression — your app is slower than your wait timeouts.

Browser-Specific Failure Analysis

import requests
import os

def get_failure_by_browser(days=7):
    """Get failure counts grouped by browser."""
    auth = (os.environ["SAUCE_USERNAME"], os.environ["SAUCE_ACCESS_KEY"])
    
    response = requests.get(
        f"https://api.us-west-1.saucelabs.com/v2/insights/trends/browsers",
        auth=auth,
        params={"time_range": f"{days}d", "status": "failed"}
    )
    
    data = response.json()
    
    for browser in data.get("browsers", []):
        print(f"{browser['browser_name']} {browser['browser_version']}: "
              f"{browser['failed']}/{browser['total']} failed "
              f"({browser['failed']/browser['total']*100:.1f}%)")

get_failure_by_browser()

Output:

safari 17: 12/45 failed (26.7%)
internet explorer 11: 8/20 failed (40.0%)
chrome latest: 2/180 failed (1.1%)
firefox latest: 1/90 failed (1.1%)

This tells you Safari and IE11 need attention.

Setting Up Automated Flakiness Alerts

Use the API to build a flakiness gate in CI:

#!/usr/bin/env python3
"""Fail CI if any test has a flakiness score above threshold."""

import os
import sys
import requests

SAUCE_USERNAME = os.environ["SAUCE_USERNAME"]
SAUCE_ACCESS_KEY = os.environ["SAUCE_ACCESS_KEY"]
FLAKINESS_THRESHOLD = 30  # Fail if any test exceeds this

def check_flakiness():
    auth = (SAUCE_USERNAME, SAUCE_ACCESS_KEY)
    response = requests.get(
        "https://api.us-west-1.saucelabs.com/v2/insights/trends/tests/flaky",
        auth=auth,
        params={"time_range": "7d", "limit": 50}
    )

    tests = response.json().get("tests", [])
    failing = [t for t in tests if t["flakiness_score"] > FLAKINESS_THRESHOLD]

    if failing:
        print(f"ERROR: {len(failing)} tests exceed flakiness threshold ({FLAKINESS_THRESHOLD}):")
        for t in failing:
            print(f"  - {t['test_name']}: score={t['flakiness_score']}")
        sys.exit(1)
    else:
        print(f"OK: All tests below flakiness threshold ({FLAKINESS_THRESHOLD})")

if __name__ == "__main__":
    check_flakiness()

Add to CI as a post-test step:

- name: Check flakiness gate
  env:
    SAUCE_USERNAME: ${{ secrets.SAUCE_USERNAME }}
    SAUCE_ACCESS_KEY: ${{ secrets.SAUCE_ACCESS_KEY }}
  run: python scripts/check_flakiness.py

Backtrace Error Reporting Integration

SauceLabs integrates with Backtrace for crash and error tracking. When a test fails with an unhandled exception, Backtrace captures the full stack trace, environment context, and groups it with similar crashes.

Setup

In SauceLabs dashboard: Settings → Integrations → Backtrace
Enter your Backtrace universe name and API token
Map SauceLabs projects to Backtrace projects

What Backtrace Captures

Full stack trace from the test session
Browser console errors during the test
Network requests that returned 4xx/5xx
SauceLabs session metadata (browser, OS, build)

Manual Error Reporting

Send custom error context during tests:

def report_error_to_backtrace(driver, error: Exception, context: dict):
    """Log error context visible in Backtrace and SauceLabs."""
    error_data = {
        "error": str(error),
        "type": type(error).__name__,
        **context
    }
    # This appears in SauceLabs session log and Backtrace
    driver.execute_script(f"sauce:context=ERROR: {error_data}")

def test_checkout(driver):
    try:
        driver.get("https://example.com/checkout")
        # ... test steps
    except Exception as e:
        report_error_to_backtrace(driver, e, {
            "step": "payment_form",
            "cart_items": 3,
            "user_type": "registered"
        })
        driver.execute_script("sauce:job-result=failed")
        raise

Reducing Test Execution Time

Insights data shows where time is spent. Common optimizations:

Parallelize by failure risk

Run tests that historically fail more often first, so CI fails fast:

# conftest.py: prioritize historically flaky tests
import os
import requests

def get_flaky_test_names():
    auth = (os.environ.get("SAUCE_USERNAME", ""), os.environ.get("SAUCE_ACCESS_KEY", ""))
    if not all(auth):
        return []
    
    r = requests.get(
        "https://api.us-west-1.saucelabs.com/v2/insights/trends/tests/flaky",
        auth=auth,
        params={"time_range": "7d", "limit": 20}
    )
    return [t["test_name"] for t in r.json().get("tests", [])]

# Use in pytest ordering plugin
FLAKY_TESTS = get_flaky_test_names()

Reuse browser sessions for independent tests

# Module-scoped driver reduces session creation overhead
@pytest.fixture(scope="module")
def driver():
    d = create_sauce_driver()
    yield d
    d.quit()

Session startup takes 10-30 seconds on SauceLabs. With 50 tests and function-scoped drivers, that's 8-25 minutes just in setup. Module scope cuts it by 5-10x.

Skip browsers for unchanged modules

def should_run_cross_browser(changed_files: list[str]) -> bool:
    """Only run cross-browser matrix if browser-relevant files changed."""
    browser_relevant = ["templates/", "static/css/", "static/js/", "views/"]
    return any(
        any(f.startswith(prefix) for prefix in browser_relevant)
        for f in changed_files
    )

Dashboard Tips

Save filtered views: Create saved filters for "flaky tests this week", "Safari failures", "builds longer than 10 minutes" — these become your daily monitoring dashboard.

Set up email digests: SauceLabs can email weekly summaries of flakiness trends. Enable under Settings → Notifications.

Use build names consistently: f"PR-{pr_number}" for PRs, f"main-{date}" for main branch. This makes the Insights timeline readable.