Selenium Grid Monitoring: Observability for Your Test Infrastructure

Selenium Grid Monitoring: Observability for Your Test Infrastructure

A Selenium Grid running without monitoring is a black box. You don't know why tests are queuing for minutes, which Node is consuming all the memory, or whether that Friday night regression actually ran against Chrome or silently fell back to nothing. Grid 4 ships with the observability primitives you need — a structured /status endpoint, a GraphQL API for deep queries, and a Prometheus-compatible metrics endpoint. This guide shows you how to wire them together into a monitoring stack that surfaces problems before they kill your CI pipeline.

What Grid 4 Exposes

Grid 4 provides three observability surfaces:

/status (HTTP GET) — A JSON snapshot of Grid health, registered Nodes, available slots, and current sessions. Fast, simple, cache-friendly. Good for health checks and basic dashboards.

/graphql (HTTP POST) — A full GraphQL API over Grid's internal data model. Query specific fields, combine node info with session details, get queue depth — all in a single request. This is the right tool for custom metrics collectors and debugging.

/prometheus (HTTP GET) — Prometheus-format metrics exposition. Exposes counters, gauges, and histograms for sessions, nodes, and request processing. Designed for scraping by Prometheus.

Grid UI — The built-in web console at the root URL. Useful for human inspection but not scriptable.

The /status Endpoint

Every Grid deployment should have /status in its health check. The response structure is consistent and tells you immediately whether the Grid can accept sessions:

curl -s http://localhost:4444/status | python3 -m json.tool
{
  "value": {
    "ready": true,
    "message": "Selenium Grid ready.",
    "nodes": [
      {
        "id": "a1b2c3d4-...",
        "uri": "http://192.168.1.20:5555",
        "maxSessions": 8,
        "availability": "UP",
        "version": "4.18.1",
        "osInfo": {
          "arch": "amd64",
          "name": "Linux",
          "version": "5.15.0"
        },
        "slots": [
          {
            "id": {"hostId": "a1b2c3d4-...", "id": "slot-1"},
            "lastStarted": "2024-02-15T10:30:45.000Z",
            "session": {
              "capabilities": {"browserName": "chrome"},
              "sessionId": "xyz789...",
              "start": "2024-02-15T10:30:45.000Z",
              "uri": "http://192.168.1.20:5555"
            },
            "stereotype": {"browserName": "chrome", "browserVersion": "120"}
          }
        ]
      }
    ]
  }
}

Parse this programmatically to track key metrics:

import requests
import json
from datetime import datetime, timezone

def get_grid_stats(grid_url: str) -> dict:
    response = requests.get(f"{grid_url}/status", timeout=10)
    data = response.json()["value"]

    nodes = data.get("nodes", [])
    total_slots = 0
    active_sessions = 0
    down_nodes = 0

    for node in nodes:
        if node["availability"] != "UP":
            down_nodes += 1
            continue
        for slot in node.get("slots", []):
            total_slots += 1
            if slot.get("session") is not None:
                active_sessions += 1

    return {
        "ready": data["ready"],
        "node_count": len(nodes),
        "down_nodes": down_nodes,
        "total_slots": total_slots,
        "active_sessions": active_sessions,
        "available_slots": total_slots - active_sessions,
        "utilization_pct": (active_sessions / total_slots * 100) if total_slots > 0 else 0
    }

stats = get_grid_stats("http://localhost:4444")
print(f"Grid ready: {stats['ready']}")
print(f"Nodes: {stats['node_count']} ({stats['down_nodes']} down)")
print(f"Sessions: {stats['active_sessions']}/{stats['total_slots']} ({stats['utilization_pct']:.1f}%)")

The GraphQL API

The /graphql endpoint gives you richer access to Grid internals. The schema covers nodes, sessions, queue state, and Grid-level summary statistics.

Basic Queries

Get session queue depth — the most important metric for detecting Grid saturation:

curl -s -X POST http://localhost:4444/graphql \
  -H "Content-Type: application/json" \
  -d '{
    "query": "{ grid { sessionCount sessionQueueSize maxSession } }"
  }'
{
  "data": {
    "grid": {
      "sessionCount": 6,
      "sessionQueueSize": 3,
      "maxSession": 20
    }
  }
}

Get detailed node status with slot utilization:

curl -s -X POST http://localhost:4444/graphql \
  -H "Content-Type: application/json" \
  -d '{
    "query": "{
      nodesInfo {
        nodes {
          id
          uri
          availability
          maxSession
          slotCount
          sessions {
            id
            capabilities
            startTime
            uri
          }
          slots {
            id
            session {
              id
              capabilities
              startTime
            }
            stereotype
          }
        }
      }
    }"
  }'

Get currently queued session requests:

curl -s -X POST http://localhost:4444/graphql \
  -H "Content-Type: application/json" \
  -d '{
    "query": "{
      sessionsInfo {
        sessionQueueRequests {
          capabilities
          requestedCapabilities
        }
      }
    }"
  }'

Python GraphQL Client

import requests
from typing import Optional

class GridGraphQL:
    def __init__(self, grid_url: str):
        self.url = f"{grid_url}/graphql"

    def query(self, q: str) -> dict:
        response = requests.post(
            self.url,
            json={"query": q},
            timeout=10
        )
        response.raise_for_status()
        result = response.json()
        if "errors" in result:
            raise ValueError(f"GraphQL errors: {result['errors']}")
        return result["data"]

    def get_queue_size(self) -> int:
        data = self.query("{ grid { sessionQueueSize } }")
        return data["grid"]["sessionQueueSize"]

    def get_node_utilization(self) -> list:
        data = self.query("""
        {
          nodesInfo {
            nodes {
              id
              uri
              availability
              maxSession
              slotCount
              sessions { id }
            }
          }
        }
        """)
        result = []
        for node in data["nodesInfo"]["nodes"]:
            active = len(node["sessions"])
            capacity = node["slotCount"] or node["maxSession"]
            result.append({
                "id": node["id"][:8],
                "uri": node["uri"],
                "availability": node["availability"],
                "active_sessions": active,
                "capacity": capacity,
                "utilization": active / capacity if capacity > 0 else 0
            })
        return result

    def get_session_age_seconds(self) -> list:
        """Returns age in seconds for all active sessions — detect hung sessions"""
        from datetime import datetime, timezone
        data = self.query("""
        {
          nodesInfo {
            nodes {
              slots {
                session {
                  id
                  startTime
                  capabilities
                }
              }
            }
          }
        }
        """)
        sessions = []
        now = datetime.now(timezone.utc)
        for node in data["nodesInfo"]["nodes"]:
            for slot in node["slots"]:
                if slot["session"]:
                    s = slot["session"]
                    start = datetime.fromisoformat(s["startTime"].replace("Z", "+00:00"))
                    age = (now - start).total_seconds()
                    sessions.append({
                        "id": s["id"][:12],
                        "age_seconds": age,
                        "browser": s["capabilities"].get("browserName", "unknown")
                    })
        return sessions

# Usage
grid = GridGraphQL("http://localhost:4444")
print(f"Queue depth: {grid.get_queue_size()}")

for node in grid.get_node_utilization():
    print(f"Node {node['id']}: {node['active_sessions']}/{node['capacity']} ({node['utilization']:.0%})")

for session in grid.get_session_age_seconds():
    if session["age_seconds"] > 180:
        print(f"WARNING: Session {session['id']} ({session['browser']}) running for {session['age_seconds']:.0f}s")

Prometheus Integration

Grid 4 exposes Prometheus metrics at /prometheus. Scrape it directly with Prometheus:

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "selenium-grid"
    static_configs:
      - targets: ["selenium-hub:4444"]
    metrics_path: /prometheus
    scrape_interval: 15s

Key metrics exposed by Grid 4:

Metric Type Description
selenium_grid_session_count Gauge Active sessions
selenium_grid_session_queue_size Gauge Sessions waiting in queue
selenium_grid_max_session Gauge Total configured session capacity
selenium_grid_node_count Gauge Registered node count
selenium_grid_up Gauge 1 if grid is ready, 0 otherwise

Kubernetes ServiceMonitor

If you're running the Prometheus Operator in Kubernetes, use a ServiceMonitor:

# selenium-grid-servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: selenium-grid
  namespace: selenium
  labels:
    app: selenium-hub
spec:
  selector:
    matchLabels:
      app: selenium-hub
  endpoints:
    - port: web
      path: /prometheus
      interval: 15s
      scheme: http

Grafana Dashboard

With Prometheus scraping Grid, build a Grafana dashboard. Here are the key panels and their PromQL queries:

Session Queue Depth (should alert if > 0 for > 60s):

selenium_grid_session_queue_size

Session Utilization (%):

(selenium_grid_session_count / selenium_grid_max_session) * 100

Available Capacity:

selenium_grid_max_session - selenium_grid_session_count

Grid Availability:

selenium_grid_up

Session Throughput (sessions started per minute, requires recording rules):

rate(selenium_grid_session_count[5m]) * 60

Dashboard JSON (condensed) for import into Grafana:

{
  "title": "Selenium Grid 4",
  "panels": [
    {
      "title": "Session Queue Depth",
      "type": "timeseries",
      "targets": [
        {
          "expr": "selenium_grid_session_queue_size",
          "legendFormat": "Queue Size"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "thresholds": {
            "steps": [
              {"color": "green", "value": 0},
              {"color": "yellow", "value": 5},
              {"color": "red", "value": 15}
            ]
          }
        }
      }
    },
    {
      "title": "Active Sessions vs Capacity",
      "type": "gauge",
      "targets": [
        {"expr": "selenium_grid_session_count", "legendFormat": "Active"},
        {"expr": "selenium_grid_max_session", "legendFormat": "Max"}
      ]
    },
    {
      "title": "Node Status",
      "type": "stat",
      "targets": [
        {
          "expr": "selenium_grid_node_count",
          "legendFormat": "Registered Nodes"
        }
      ]
    }
  ]
}

Alerting Rules

Define Prometheus alerting rules for the scenarios that break test pipelines:

# selenium-alerts.yaml
groups:
  - name: selenium.grid
    interval: 30s
    rules:

      # Grid is down
      - alert: SeleniumGridDown
        expr: selenium_grid_up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Selenium Grid is not ready"
          description: "The Grid has been reporting not-ready for more than 1 minute. All test sessions will fail."

      # Session queue backing up
      - alert: SeleniumSessionQueueHigh
        expr: selenium_grid_session_queue_size > 10
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Selenium Grid session queue is high ({{ $value }} sessions waiting)"
          description: "More than 10 sessions have been queued for over 2 minutes. Consider scaling up nodes."

      # Queue non-zero for too long (tests will time out)
      - alert: SeleniumSessionQueueStuck
        expr: selenium_grid_session_queue_size > 0
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Selenium Grid session queue stuck"
          description: "Sessions have been queued for 10+ minutes. Tests are likely timing out."

      # Grid at full capacity
      - alert: SeleniumGridAtCapacity
        expr: |
          (selenium_grid_session_count / selenium_grid_max_session) >= 0.95
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Selenium Grid near capacity ({{ $value | humanizePercentage }} utilized)"
          description: "Grid has been above 95% session capacity for 5 minutes. New sessions will queue."

      # No nodes registered (all nodes crashed)
      - alert: SeleniumNoNodes
        expr: selenium_grid_node_count == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "No Selenium nodes registered"
          description: "The Grid has no registered nodes. All session requests will fail immediately."

Apply to Prometheus:

# prometheus.yml (add to rule_files)
rule_files:
  - "selenium-alerts.yaml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

Log Aggregation

Grid 4 produces structured JSON logs when run with --log-level INFO (default) or --log-level FINE (verbose). In Docker/Kubernetes, these go to stdout and get collected by your container log driver.

Configure log format for machine parsing:

java -jar selenium-server-4.18.1.jar hub \
  --log-level INFO \
  2>&1 | tee grid-hub.log

For Kubernetes with Fluentd or Fluent Bit, add a log parser for Grid's output format:

# fluent-bit ConfigMap excerpt
[PARSER]
    Name        selenium-grid
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z

[FILTER]
    Name        parser
    Match       selenium.*
    Key_Name    log
    Parser      selenium-grid
    Reserve_Data True

Node Failure Detection via Logs

Node failures manifest as specific log patterns. Monitor for these strings in your log aggregation system:

# Node registered:
"Registered with distributor"

# Node disconnected (network issue or crash):
"Lost connection to the event bus"

# Session creation failure:
"Unable to create session"

# Slot timeout (test hung, session killed):
"Session has been running for too long"

# Browser crash:
"Session timed out or not found"

A simple log-based alert in a system like Loki+Grafana:

# Alert if node disconnection rate exceeds 3/minute
count_over_time({app="selenium-node-chrome"} |= "Lost connection to the event bus" [1m]) > 3

Custom Metrics Exporter

For teams that need metrics Grid doesn't expose natively (per-browser session counts, individual Node utilization, session age distribution), build a lightweight exporter:

#!/usr/bin/env python3
"""
selenium-grid-exporter.py
Exports Grid 4 metrics to Prometheus format via /metrics endpoint
"""

import time
import requests
from prometheus_client import start_http_server, Gauge, Counter, REGISTRY
from prometheus_client.core import GaugeMetricFamily, CounterMetricFamily

GRID_URL = "http://localhost:4444"

class GridCollector:
    def collect(self):
        try:
            # /status for node-level data
            status = requests.get(f"{GRID_URL}/status", timeout=5).json()["value"]

            # /graphql for queue and session data
            graphql_response = requests.post(
                f"{GRID_URL}/graphql",
                json={"query": "{ grid { sessionCount sessionQueueSize maxSession nodeCount } }"},
                timeout=5
            ).json()
            grid_data = graphql_response["data"]["grid"]

        except Exception as e:
            print(f"Collection error: {e}")
            return

        # Grid-level metrics
        yield GaugeMetricFamily(
            "selenium_grid_ready",
            "Whether the grid is ready to accept sessions",
            value=1 if status.get("ready") else 0
        )

        yield GaugeMetricFamily(
            "selenium_grid_session_queue_size",
            "Number of sessions waiting for a node",
            value=grid_data["sessionQueueSize"]
        )

        yield GaugeMetricFamily(
            "selenium_grid_active_sessions",
            "Number of active sessions",
            value=grid_data["sessionCount"]
        )

        yield GaugeMetricFamily(
            "selenium_grid_max_sessions",
            "Maximum sessions the grid can run simultaneously",
            value=grid_data["maxSession"]
        )

        # Per-node metrics
        node_active = GaugeMetricFamily(
            "selenium_grid_node_active_sessions",
            "Active sessions per node",
            labels=["node_id", "node_uri", "availability"]
        )
        node_capacity = GaugeMetricFamily(
            "selenium_grid_node_capacity",
            "Max sessions per node",
            labels=["node_id", "node_uri"]
        )

        for node in status.get("nodes", []):
            node_id = node["id"][:8]
            node_uri = node["uri"]
            availability = node["availability"]
            slots = node.get("slots", [])
            active = sum(1 for s in slots if s.get("session") is not None)

            node_active.add_metric(
                [node_id, node_uri, availability],
                active
            )
            node_capacity.add_metric(
                [node_id, node_uri],
                node["maxSessions"]
            )

        yield node_active
        yield node_capacity


if __name__ == "__main__":
    REGISTRY.register(GridCollector())
    start_http_server(9090)
    print("Grid exporter running on :9090/metrics")
    while True:
        time.sleep(30)
pip install prometheus-client requests
python3 selenium-grid-exporter.py &

# Verify
curl -s http://localhost:9090/metrics | grep selenium_grid
# selenium_grid_ready 1.0
# selenium_grid_session_queue_size 0.0
# selenium_grid_active_sessions 3.0
# selenium_grid_max_sessions 20.0
# selenium_grid_node_active_sessions{availability="UP",node_id="a1b2c3d4",node_uri="http://..."} 3.0

Grid UI as a Debugging Tool

The built-in Grid UI (http://localhost:4444) provides a visual representation of Node status and active sessions that's useful for ad-hoc debugging. But for production observability, the UI has limits: it auto-refreshes every few seconds (not real-time), doesn't retain history, and can't be queried programmatically.

Use the UI for: diagnosing stuck sessions, verifying Node registration, checking capability mismatches during initial setup.

Use Prometheus+Grafana for: alerting, trend analysis, capacity planning, post-incident review.

Detecting and Handling Hung Sessions

The most common Grid operational problem is sessions that don't call driver.quit() — leaked sessions hold slots indefinitely. Grid's session-timeout setting (SE_NODE_SESSION_TIMEOUT) kills idle sessions after N seconds. But combine this with active monitoring:

# hung-session-detector.py — run as a cron job or daemon
import requests
from datetime import datetime, timezone

GRID_URL = "http://localhost:4444"
MAX_SESSION_AGE_SECONDS = 600  # 10 minutes — adjust to your test duration

query = """
{
  nodesInfo {
    nodes {
      slots {
        session {
          id
          startTime
          capabilities
        }
      }
    }
  }
}
"""

response = requests.post(f"{GRID_URL}/graphql", json={"query": query})
data = response.json()["data"]
now = datetime.now(timezone.utc)

for node in data["nodesInfo"]["nodes"]:
    for slot in node["slots"]:
        session = slot.get("session")
        if not session:
            continue

        start = datetime.fromisoformat(session["startTime"].replace("Z", "+00:00"))
        age = (now - start).total_seconds()

        if age > MAX_SESSION_AGE_SECONDS:
            print(f"HUNG SESSION DETECTED:")
            print(f"  ID: {session['id']}")
            print(f"  Browser: {session['capabilities'].get('browserName')}")
            print(f"  Running for: {age:.0f}s")
            print(f"  DELETE: DELETE http://{GRID_URL}/session/{session['id']}")

Combined — the Grid 4 observability stack (structured /status, the GraphQL API, Prometheus scraping, Grafana dashboards, and alerting rules) gives you complete visibility into what your test infrastructure is doing. Teams that invest in this monitoring layer spend far less time debugging mysterious test failures caused by infrastructure problems and far more time improving the tests themselves.

Read more

Start now free