Splunk Synthetic Monitoring: Browser Tests, Uptime Checks, and Waterfall Analysis

Before users report a broken checkout or a slow login page, you want to know about it. Synthetic monitoring is the discipline of simulating user interactions from external locations on a continuous schedule — confirming that your application works correctly even when no real users are actively using it.

Splunk Synthetic Monitoring (formerly Rigor, acquired by Splunk in 2020) is one of the most capable platforms in this category. This guide covers how it works, what types of monitors it supports, how to interpret waterfall analysis, and how to wire up alerts that actually page the right people.

What Is Splunk Synthetic Monitoring?

Splunk Synthetic Monitoring runs scripted checks against your application from managed probe locations around the world. It operates entirely outside your network, simulating real user requests through real browser engines — catching issues that internal health checks miss.

It sits within the Splunk Observability Cloud umbrella, which means synthetic test results are correlated with metrics, traces, and logs from other Splunk products. When a browser test fails, you can jump directly into APM to see what happened on the backend during that test run.

There are three primary monitor types:

1. Browser Tests — full Chromium-based scripted flows
2. Uptime Tests (formerly API checks in Rigor) — HTTP(S) endpoint checks
3. API Tests — multi-step API workflows with assertions

Browser Tests

Browser tests are the flagship feature of Splunk Synthetic Monitoring. They run a real Chromium browser, execute JavaScript, load external resources, and measure performance metrics that matter to users: Time to First Byte, First Contentful Paint, Largest Contentful Paint, Cumulative Layout Shift, Total Blocking Time.

Writing Browser Test Scripts

Scripts are written in JavaScript using a Selenium-inspired API. A simple login flow looks like:

// Navigate to login page
await driver.get('https://app.example.com/login');

// Wait for email field and enter credentials
await driver.findElement(By.css('#email')).sendKeys('monitor@example.com');
await driver.findElement(By.css('#password')).sendKeys(process.env.MONITOR_PASSWORD);
await driver.findElement(By.css('button[type="submit"]')).click();

// Assert successful login by checking for dashboard element
await driver.wait(until.elementLocated(By.css('.dashboard-header')), 5000);
const headerText = await driver.findElement(By.css('.dashboard-header')).getText();
assert.include(headerText, 'Welcome');

Splunk's editor provides a point-and-click recorder that generates this JavaScript automatically by watching your browser interactions — useful for non-developers who need to set up basic flows.

Sensitive Data Handling

Test credentials and API keys should never be hardcoded in scripts. Splunk Synthetic Monitoring supports Global Variables — encrypted key-value pairs stored in the platform and referenced via process.env.VARIABLE_NAME in scripts. Variable values are masked in logs and screenshots.

Performance Metrics Captured

Every browser test run captures a full performance profile:

• TTFB (Time to First Byte) — server response time
• FCP (First Contentful Paint) — when first content renders
• LCP (Largest Contentful Paint) — when the largest content element renders (Core Web Vital)
• CLS (Cumulative Layout Shift) — visual stability score (Core Web Vital)
• TBT (Total Blocking Time) — sum of long tasks blocking the main thread
• Page Load Time — total time to fully load the page
• DOM Interactive — when the browser can begin processing interactions

These are surfaced in both the individual run view and aggregated trend charts over time.

Waterfall Analysis

The waterfall chart is where Splunk Synthetic really earns its keep for performance debugging. For every browser test run, it renders the full network waterfall: every HTTP request made by the page, its start time, duration, and HTTP status code.

Reading the Waterfall

Each row in the waterfall represents one network request. The bar spans the request's timeline, color-coded by phase:

• DNS (dark green) — DNS resolution time
• Connect (orange) — TCP connection establishment
• SSL (purple) — TLS handshake
• Send (light blue) — time to send the request
• Wait (blue) — server processing time (TTFB for this resource)
• Receive (light green) — download time

Common patterns to look for:

Long DNS times on some resources but not others often indicate third-party scripts (analytics, chat widgets, A/B testing tools) resolving from different nameservers. These are frequently overlooked performance killers.

Render-blocking resources appear as long horizontal chains early in the waterfall where no other requests start until they finish. Classic cause: synchronous <script> tags in <head> loading large JavaScript files before the browser can render anything.

Waterfall stairstepping — a cascading pattern where requests start one at a time instead of in parallel — often indicates resources discovered late in parsing (via CSS @import or dynamically injected scripts) or HTTP/1.1 connection limits.

Filmstrip View

Alongside the waterfall, Splunk Synthetic provides a filmstrip: screenshots of the page captured at regular intervals during load. This is invaluable for diagnosing layout shift and identifying when meaningful content actually appears to users versus when the technical "load complete" event fires.

Comparing filmstrips between runs before and after a deployment quickly reveals visual regressions.

Uptime Tests

Uptime tests are lightweight HTTP(S) checks — no browser involved, just a raw HTTP request to a URL with configurable assertions.

Configuration Options

• Method: GET, POST, PUT, DELETE, HEAD
• Request headers: Set Authorization, Content-Type, custom headers
• Request body: JSON, form data, or raw
• Assertions:
  • Status code (exact match or range: 2xx, 3xx)
  • Response time threshold
  • Response body contains string
  • Response body matches regex
  • Response header contains value
  • SSL certificate expiry threshold

A typical uptime check for an API health endpoint:

URL: https://api.example.com/health
Method: GET
Headers: Authorization: Bearer ${HEALTH_CHECK_TOKEN}
Assertions:
  - Status code: 200
  - Response time: < 500ms
  - Body contains: "status":"ok"
  - SSL cert expires: > 14 days

Probe Locations

Splunk Synthetic runs checks from 50+ global probe locations including North America, Europe, Asia-Pacific, South America, and the Middle East. You can run a single monitor from all locations or select a specific subset.

Running from multiple locations simultaneously catches regional failures — CDN misconfiguration, DNS propagation issues, or data center problems that only affect users in specific geographies.

API Tests

API tests extend uptime checks into multi-step workflows. Each step can extract values from the previous step's response and pass them to the next. This enables testing authenticated API flows end to end.

Example: OAuth Token + API Call

Step 1 — Get Token
  POST https://auth.example.com/oauth/token
  Body: grant_type=client_credentials&client_id=${CLIENT_ID}&client_secret=${CLIENT_SECRET}
  Extract: access_token → {{token}}

Step 2 — Create Resource
  POST https://api.example.com/v1/orders
  Headers: Authorization: Bearer {{token}}
  Body: {"product_id": "prod_123", "quantity": 1}
  Assert: status 201
  Extract: id → {{order_id}}

Step 3 — Verify Resource
  GET https://api.example.com/v1/orders/{{order_id}}
  Headers: Authorization: Bearer {{token}}
  Assert: status 200
  Assert: body contains "prod_123"

This pattern is essential for testing APIs that require authentication — which is most of them.

Alerting

Splunk Synthetic Monitoring supports flexible alerting with per-monitor thresholds.

Alert Conditions

For browser tests:

• Test failure (assertion failed, element not found, script error)
• Performance threshold exceeded (e.g., LCP > 3000ms)
• Failure from N locations simultaneously (reduces false positives)
• Consecutive failures (fire only after 2 failures in a row)

For uptime and API tests:

• Status code mismatch
• Assertion failed
• Response time exceeded
• SSL expiry approaching

Notification Channels

Alerts route to the same notification channels as Splunk Observability Cloud detectors: PagerDuty, OpsGenie, Slack, email, and webhooks. If you're using Splunk On-Call, synthetic failures feed into the same incident management workflow as infrastructure and APM alerts.

Alert Tuning

The most common mistake with synthetic monitoring is alerting on single-probe, single-failure events. Network hiccups from probe infrastructure cause occasional false positives. Best practice:

• Require failure from at least 2 probe locations before alerting
• Require at least 2 consecutive failures before firing
• Use lower severity for performance threshold breaches vs. complete failures

Correlating Synthetic Results with Backend Observability

When a browser test fails or shows degraded performance, the root cause is usually in the backend. Splunk's integration between Synthetic Monitoring and APM enables direct correlation:

1. Browser test detects high LCP
2. Click "View in APM" from the synthetic test result
3. APM shows the traces for backend requests made during the test run
4. A slow database query in the checkout service is responsible for the 3s response time

This pivot from frontend symptom to backend cause, in a few clicks, is the key advantage of using Splunk Synthetic as part of the broader Observability Cloud rather than as a standalone tool.

Synthetic vs. Real User Monitoring (RUM)

Splunk also offers RUM — JavaScript injected into your pages that captures performance data from actual user sessions. The two are complementary:

Synthetic catches issues before users encounter them. RUM tells you what users actually experienced. Both together give full coverage.

Where HelpMeTest Fits

Splunk Synthetic Monitoring covers infrastructure-level user flow verification excellently. For teams who also need to run functional regression tests — validating that specific business logic, form validations, or multi-user workflows behave correctly — HelpMeTest adds a natural layer. Write tests in plain English, run them on every deployment, and get pass/fail confirmation before Splunk Synthetic's scheduled checks even run.

Getting Started with Splunk Synthetic Monitoring

1. Start with uptime tests — add your most critical endpoints in under 5 minutes
2. Set up a login browser test — the highest-value flow to monitor continuously
3. Run from 3+ global locations — catch regional issues immediately
4. Configure conservative alerting — require 2 consecutive failures from 2 locations
5. Connect to Splunk APM — enable the backend correlation workflow

Splunk Synthetic Monitoring's strength is depth: waterfall analysis, filmstrip comparison, multi-step API flows, and tight integration with the rest of the Splunk observability stack. For teams running serious production workloads, it's one of the most thorough synthetic monitoring solutions available.