Selenium Grid Monitoring: Observability for Your Test Infrastructure
A Selenium Grid running without monitoring is a black box. You don't know why tests are queuing for minutes, which Node is consuming all the memory, or whether that Friday night regression actually ran against Chrome or silently fell back to nothing. Grid 4 ships with the observability primitives you need — a structured /status endpoint, a GraphQL API for deep queries, and a Prometheus-compatible metrics endpoint. This guide shows you how to wire them together into a monitoring stack that surfaces problems before they kill your CI pipeline.
What Grid 4 Exposes
Grid 4 provides three observability surfaces:
/status (HTTP GET) — A JSON snapshot of Grid health, registered Nodes, available slots, and current sessions. Fast, simple, cache-friendly. Good for health checks and basic dashboards.
/graphql (HTTP POST) — A full GraphQL API over Grid's internal data model. Query specific fields, combine node info with session details, get queue depth — all in a single request. This is the right tool for custom metrics collectors and debugging.
/prometheus (HTTP GET) — Prometheus-format metrics exposition. Exposes counters, gauges, and histograms for sessions, nodes, and request processing. Designed for scraping by Prometheus.
Grid UI — The built-in web console at the root URL. Useful for human inspection but not scriptable.
The /status Endpoint
Every Grid deployment should have /status in its health check. The response structure is consistent and tells you immediately whether the Grid can accept sessions:
curl -s http://localhost:4444/status | python3 -m json.tool{
"value": {
"ready": true,
"message": "Selenium Grid ready.",
"nodes": [
{
"id": "a1b2c3d4-...",
"uri": "http://192.168.1.20:5555",
"maxSessions": 8,
"availability": "UP",
"version": "4.18.1",
"osInfo": {
"arch": "amd64",
"name": "Linux",
"version": "5.15.0"
},
"slots": [
{
"id": {"hostId": "a1b2c3d4-...", "id": "slot-1"},
"lastStarted": "2024-02-15T10:30:45.000Z",
"session": {
"capabilities": {"browserName": "chrome"},
"sessionId": "xyz789...",
"start": "2024-02-15T10:30:45.000Z",
"uri": "http://192.168.1.20:5555"
},
"stereotype": {"browserName": "chrome", "browserVersion": "120"}
}
]
}
]
}
}Parse this programmatically to track key metrics:
import requests
import json
from datetime import datetime, timezone
def get_grid_stats(grid_url: str) -> dict:
response = requests.get(f"{grid_url}/status", timeout=10)
data = response.json()["value"]
nodes = data.get("nodes", [])
total_slots = 0
active_sessions = 0
down_nodes = 0
for node in nodes:
if node["availability"] != "UP":
down_nodes += 1
continue
for slot in node.get("slots", []):
total_slots += 1
if slot.get("session") is not None:
active_sessions += 1
return {
"ready": data["ready"],
"node_count": len(nodes),
"down_nodes": down_nodes,
"total_slots": total_slots,
"active_sessions": active_sessions,
"available_slots": total_slots - active_sessions,
"utilization_pct": (active_sessions / total_slots * 100) if total_slots > 0 else 0
}
stats = get_grid_stats("http://localhost:4444")
print(f"Grid ready: {stats['ready']}")
print(f"Nodes: {stats['node_count']} ({stats['down_nodes']} down)")
print(f"Sessions: {stats['active_sessions']}/{stats['total_slots']} ({stats['utilization_pct']:.1f}%)")The GraphQL API
The /graphql endpoint gives you richer access to Grid internals. The schema covers nodes, sessions, queue state, and Grid-level summary statistics.
Basic Queries
Get session queue depth — the most important metric for detecting Grid saturation:
curl -s -X POST http://localhost:4444/graphql \
-H "Content-Type: application/json" \
-d '{
"query": "{ grid { sessionCount sessionQueueSize maxSession } }"
}'{
"data": {
"grid": {
"sessionCount": 6,
"sessionQueueSize": 3,
"maxSession": 20
}
}
}Get detailed node status with slot utilization:
curl -s -X POST http://localhost:4444/graphql \
-H "Content-Type: application/json" \
-d '{
"query": "{
nodesInfo {
nodes {
id
uri
availability
maxSession
slotCount
sessions {
id
capabilities
startTime
uri
}
slots {
id
session {
id
capabilities
startTime
}
stereotype
}
}
}
}"
}'Get currently queued session requests:
curl -s -X POST http://localhost:4444/graphql \
-H "Content-Type: application/json" \
-d '{
"query": "{
sessionsInfo {
sessionQueueRequests {
capabilities
requestedCapabilities
}
}
}"
}'Python GraphQL Client
import requests
from typing import Optional
class GridGraphQL:
def __init__(self, grid_url: str):
self.url = f"{grid_url}/graphql"
def query(self, q: str) -> dict:
response = requests.post(
self.url,
json={"query": q},
timeout=10
)
response.raise_for_status()
result = response.json()
if "errors" in result:
raise ValueError(f"GraphQL errors: {result['errors']}")
return result["data"]
def get_queue_size(self) -> int:
data = self.query("{ grid { sessionQueueSize } }")
return data["grid"]["sessionQueueSize"]
def get_node_utilization(self) -> list:
data = self.query("""
{
nodesInfo {
nodes {
id
uri
availability
maxSession
slotCount
sessions { id }
}
}
}
""")
result = []
for node in data["nodesInfo"]["nodes"]:
active = len(node["sessions"])
capacity = node["slotCount"] or node["maxSession"]
result.append({
"id": node["id"][:8],
"uri": node["uri"],
"availability": node["availability"],
"active_sessions": active,
"capacity": capacity,
"utilization": active / capacity if capacity > 0 else 0
})
return result
def get_session_age_seconds(self) -> list:
"""Returns age in seconds for all active sessions — detect hung sessions"""
from datetime import datetime, timezone
data = self.query("""
{
nodesInfo {
nodes {
slots {
session {
id
startTime
capabilities
}
}
}
}
}
""")
sessions = []
now = datetime.now(timezone.utc)
for node in data["nodesInfo"]["nodes"]:
for slot in node["slots"]:
if slot["session"]:
s = slot["session"]
start = datetime.fromisoformat(s["startTime"].replace("Z", "+00:00"))
age = (now - start).total_seconds()
sessions.append({
"id": s["id"][:12],
"age_seconds": age,
"browser": s["capabilities"].get("browserName", "unknown")
})
return sessions
# Usage
grid = GridGraphQL("http://localhost:4444")
print(f"Queue depth: {grid.get_queue_size()}")
for node in grid.get_node_utilization():
print(f"Node {node['id']}: {node['active_sessions']}/{node['capacity']} ({node['utilization']:.0%})")
for session in grid.get_session_age_seconds():
if session["age_seconds"] > 180:
print(f"WARNING: Session {session['id']} ({session['browser']}) running for {session['age_seconds']:.0f}s")Prometheus Integration
Grid 4 exposes Prometheus metrics at /prometheus. Scrape it directly with Prometheus:
# prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: "selenium-grid"
static_configs:
- targets: ["selenium-hub:4444"]
metrics_path: /prometheus
scrape_interval: 15sKey metrics exposed by Grid 4:
| Metric | Type | Description |
|---|---|---|
selenium_grid_session_count |
Gauge | Active sessions |
selenium_grid_session_queue_size |
Gauge | Sessions waiting in queue |
selenium_grid_max_session |
Gauge | Total configured session capacity |
selenium_grid_node_count |
Gauge | Registered node count |
selenium_grid_up |
Gauge | 1 if grid is ready, 0 otherwise |
Kubernetes ServiceMonitor
If you're running the Prometheus Operator in Kubernetes, use a ServiceMonitor:
# selenium-grid-servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: selenium-grid
namespace: selenium
labels:
app: selenium-hub
spec:
selector:
matchLabels:
app: selenium-hub
endpoints:
- port: web
path: /prometheus
interval: 15s
scheme: httpGrafana Dashboard
With Prometheus scraping Grid, build a Grafana dashboard. Here are the key panels and their PromQL queries:
Session Queue Depth (should alert if > 0 for > 60s):
selenium_grid_session_queue_sizeSession Utilization (%):
(selenium_grid_session_count / selenium_grid_max_session) * 100Available Capacity:
selenium_grid_max_session - selenium_grid_session_countGrid Availability:
selenium_grid_upSession Throughput (sessions started per minute, requires recording rules):
rate(selenium_grid_session_count[5m]) * 60Dashboard JSON (condensed) for import into Grafana:
{
"title": "Selenium Grid 4",
"panels": [
{
"title": "Session Queue Depth",
"type": "timeseries",
"targets": [
{
"expr": "selenium_grid_session_queue_size",
"legendFormat": "Queue Size"
}
],
"fieldConfig": {
"defaults": {
"thresholds": {
"steps": [
{"color": "green", "value": 0},
{"color": "yellow", "value": 5},
{"color": "red", "value": 15}
]
}
}
}
},
{
"title": "Active Sessions vs Capacity",
"type": "gauge",
"targets": [
{"expr": "selenium_grid_session_count", "legendFormat": "Active"},
{"expr": "selenium_grid_max_session", "legendFormat": "Max"}
]
},
{
"title": "Node Status",
"type": "stat",
"targets": [
{
"expr": "selenium_grid_node_count",
"legendFormat": "Registered Nodes"
}
]
}
]
}Alerting Rules
Define Prometheus alerting rules for the scenarios that break test pipelines:
# selenium-alerts.yaml
groups:
- name: selenium.grid
interval: 30s
rules:
# Grid is down
- alert: SeleniumGridDown
expr: selenium_grid_up == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Selenium Grid is not ready"
description: "The Grid has been reporting not-ready for more than 1 minute. All test sessions will fail."
# Session queue backing up
- alert: SeleniumSessionQueueHigh
expr: selenium_grid_session_queue_size > 10
for: 2m
labels:
severity: warning
annotations:
summary: "Selenium Grid session queue is high ({{ $value }} sessions waiting)"
description: "More than 10 sessions have been queued for over 2 minutes. Consider scaling up nodes."
# Queue non-zero for too long (tests will time out)
- alert: SeleniumSessionQueueStuck
expr: selenium_grid_session_queue_size > 0
for: 10m
labels:
severity: critical
annotations:
summary: "Selenium Grid session queue stuck"
description: "Sessions have been queued for 10+ minutes. Tests are likely timing out."
# Grid at full capacity
- alert: SeleniumGridAtCapacity
expr: |
(selenium_grid_session_count / selenium_grid_max_session) >= 0.95
for: 5m
labels:
severity: warning
annotations:
summary: "Selenium Grid near capacity ({{ $value | humanizePercentage }} utilized)"
description: "Grid has been above 95% session capacity for 5 minutes. New sessions will queue."
# No nodes registered (all nodes crashed)
- alert: SeleniumNoNodes
expr: selenium_grid_node_count == 0
for: 2m
labels:
severity: critical
annotations:
summary: "No Selenium nodes registered"
description: "The Grid has no registered nodes. All session requests will fail immediately."Apply to Prometheus:
# prometheus.yml (add to rule_files)
rule_files:
- "selenium-alerts.yaml"
alerting:
alertmanagers:
- static_configs:
- targets: ["alertmanager:9093"]Log Aggregation
Grid 4 produces structured JSON logs when run with --log-level INFO (default) or --log-level FINE (verbose). In Docker/Kubernetes, these go to stdout and get collected by your container log driver.
Configure log format for machine parsing:
java -jar selenium-server-4.18.1.jar hub \
--log-level INFO \
2>&1 | tee grid-hub.logFor Kubernetes with Fluentd or Fluent Bit, add a log parser for Grid's output format:
# fluent-bit ConfigMap excerpt
[PARSER]
Name selenium-grid
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
[FILTER]
Name parser
Match selenium.*
Key_Name log
Parser selenium-grid
Reserve_Data TrueNode Failure Detection via Logs
Node failures manifest as specific log patterns. Monitor for these strings in your log aggregation system:
# Node registered:
"Registered with distributor"
# Node disconnected (network issue or crash):
"Lost connection to the event bus"
# Session creation failure:
"Unable to create session"
# Slot timeout (test hung, session killed):
"Session has been running for too long"
# Browser crash:
"Session timed out or not found"A simple log-based alert in a system like Loki+Grafana:
# Alert if node disconnection rate exceeds 3/minute
count_over_time({app="selenium-node-chrome"} |= "Lost connection to the event bus" [1m]) > 3Custom Metrics Exporter
For teams that need metrics Grid doesn't expose natively (per-browser session counts, individual Node utilization, session age distribution), build a lightweight exporter:
#!/usr/bin/env python3
"""
selenium-grid-exporter.py
Exports Grid 4 metrics to Prometheus format via /metrics endpoint
"""
import time
import requests
from prometheus_client import start_http_server, Gauge, Counter, REGISTRY
from prometheus_client.core import GaugeMetricFamily, CounterMetricFamily
GRID_URL = "http://localhost:4444"
class GridCollector:
def collect(self):
try:
# /status for node-level data
status = requests.get(f"{GRID_URL}/status", timeout=5).json()["value"]
# /graphql for queue and session data
graphql_response = requests.post(
f"{GRID_URL}/graphql",
json={"query": "{ grid { sessionCount sessionQueueSize maxSession nodeCount } }"},
timeout=5
).json()
grid_data = graphql_response["data"]["grid"]
except Exception as e:
print(f"Collection error: {e}")
return
# Grid-level metrics
yield GaugeMetricFamily(
"selenium_grid_ready",
"Whether the grid is ready to accept sessions",
value=1 if status.get("ready") else 0
)
yield GaugeMetricFamily(
"selenium_grid_session_queue_size",
"Number of sessions waiting for a node",
value=grid_data["sessionQueueSize"]
)
yield GaugeMetricFamily(
"selenium_grid_active_sessions",
"Number of active sessions",
value=grid_data["sessionCount"]
)
yield GaugeMetricFamily(
"selenium_grid_max_sessions",
"Maximum sessions the grid can run simultaneously",
value=grid_data["maxSession"]
)
# Per-node metrics
node_active = GaugeMetricFamily(
"selenium_grid_node_active_sessions",
"Active sessions per node",
labels=["node_id", "node_uri", "availability"]
)
node_capacity = GaugeMetricFamily(
"selenium_grid_node_capacity",
"Max sessions per node",
labels=["node_id", "node_uri"]
)
for node in status.get("nodes", []):
node_id = node["id"][:8]
node_uri = node["uri"]
availability = node["availability"]
slots = node.get("slots", [])
active = sum(1 for s in slots if s.get("session") is not None)
node_active.add_metric(
[node_id, node_uri, availability],
active
)
node_capacity.add_metric(
[node_id, node_uri],
node["maxSessions"]
)
yield node_active
yield node_capacity
if __name__ == "__main__":
REGISTRY.register(GridCollector())
start_http_server(9090)
print("Grid exporter running on :9090/metrics")
while True:
time.sleep(30)pip install prometheus-client requests
python3 selenium-grid-exporter.py &
# Verify
curl -s http://localhost:9090/metrics | grep selenium_grid
# selenium_grid_ready 1.0
# selenium_grid_session_queue_size 0.0
# selenium_grid_active_sessions 3.0
# selenium_grid_max_sessions 20.0
# selenium_grid_node_active_sessions{availability="UP",node_id="a1b2c3d4",node_uri="http://..."} 3.0Grid UI as a Debugging Tool
The built-in Grid UI (http://localhost:4444) provides a visual representation of Node status and active sessions that's useful for ad-hoc debugging. But for production observability, the UI has limits: it auto-refreshes every few seconds (not real-time), doesn't retain history, and can't be queried programmatically.
Use the UI for: diagnosing stuck sessions, verifying Node registration, checking capability mismatches during initial setup.
Use Prometheus+Grafana for: alerting, trend analysis, capacity planning, post-incident review.
Detecting and Handling Hung Sessions
The most common Grid operational problem is sessions that don't call driver.quit() — leaked sessions hold slots indefinitely. Grid's session-timeout setting (SE_NODE_SESSION_TIMEOUT) kills idle sessions after N seconds. But combine this with active monitoring:
# hung-session-detector.py — run as a cron job or daemon
import requests
from datetime import datetime, timezone
GRID_URL = "http://localhost:4444"
MAX_SESSION_AGE_SECONDS = 600 # 10 minutes — adjust to your test duration
query = """
{
nodesInfo {
nodes {
slots {
session {
id
startTime
capabilities
}
}
}
}
}
"""
response = requests.post(f"{GRID_URL}/graphql", json={"query": query})
data = response.json()["data"]
now = datetime.now(timezone.utc)
for node in data["nodesInfo"]["nodes"]:
for slot in node["slots"]:
session = slot.get("session")
if not session:
continue
start = datetime.fromisoformat(session["startTime"].replace("Z", "+00:00"))
age = (now - start).total_seconds()
if age > MAX_SESSION_AGE_SECONDS:
print(f"HUNG SESSION DETECTED:")
print(f" ID: {session['id']}")
print(f" Browser: {session['capabilities'].get('browserName')}")
print(f" Running for: {age:.0f}s")
print(f" DELETE: DELETE http://{GRID_URL}/session/{session['id']}")Combined — the Grid 4 observability stack (structured /status, the GraphQL API, Prometheus scraping, Grafana dashboards, and alerting rules) gives you complete visibility into what your test infrastructure is doing. Teams that invest in this monitoring layer spend far less time debugging mysterious test failures caused by infrastructure problems and far more time improving the tests themselves.