Dynatrace Davis AI: Anomaly Detection and Root Cause Analysis Explained

Most monitoring systems are alarm systems — they fire when a metric crosses a threshold you configured weeks ago, often incorrectly. Dynatrace Davis AI is something different: it learns what normal looks like for every entity in your environment, detects deviations, correlates them across your entire stack, and names a root cause. This post explains exactly how Davis works, what it gets right, where it has limits, and how to configure it for real-world production environments.

The Core Problem Davis Solves

Imagine you have 200 microservices. A typical threshold-based APM setup requires someone to define alert thresholds for each service. That's 200 response time thresholds, 200 error rate thresholds, 200 CPU thresholds — hundreds or thousands of rules, most of which are wrong because they were set at deployment time and never updated as traffic patterns changed.

When something breaks, you get 47 alerts firing simultaneously. You spend 90 minutes figuring out which alert represents the cause and which 46 are symptoms.

Davis inverts this. It:

1. Learns baselines automatically — no thresholds to configure
2. Detects deviations as they happen
3. Groups correlated events into a single Problem
4. Identifies the root cause entity and the triggering event

Instead of 47 alerts, you get one Problem card that says: "Checkout service degraded. Root cause: database connection pool exhausted on db-prod-02. Started 14 minutes ago. 312 users affected."

How Davis Learns Baselines

Davis uses a technique called automatic adaptive baselining. For every monitored metric (response time, error rate, throughput, CPU, memory), Davis analyzes historical data to model:

• Hourly patterns: traffic is higher at 9am than 3am
• Day-of-week patterns: weekday traffic differs from weekend
• Seasonal trends: end-of-month spikes, holiday traffic changes
• Sudden shifts: if your app gains 10x more users, the baseline adjusts over days

Davis builds separate baselines for each entity (each service, each host, each process). A spike in response time that is normal for your batch processing service is abnormal for your real-time checkout service — Davis knows the difference.

The baseline model requires roughly 7 days of data to stabilize. In the first week after deploying Dynatrace, expect some false positives.

Anomaly Detection: What Davis Watches

Davis monitors anomalies across multiple signal types simultaneously:

Service-level anomalies

• Response time degradation: p50, p90, p99 significantly above baseline
• Error rate spike: percentage of failed requests above normal
• Throughput drop: sudden reduction in request rate (often a sign of upstream failure)
• Failure rate increase: requests returning 5xx codes

Infrastructure anomalies

• CPU saturation: sustained high CPU on a host or container
• Memory pressure: available memory dropping, GC pressure in JVM
• Network errors: packet loss, connection refused, DNS failures
• Disk I/O: read/write latency spikes

Process and runtime anomalies

• JVM GC pause: garbage collection pauses exceeding thresholds
• Thread pool exhaustion: available threads approaching zero
• Connection pool saturation: database or HTTP connection pools maxed out
• Process crashes: unexpected process restarts

External dependency anomalies

• Third-party service slowness: calls to external APIs degrading
• CDN issues: specific resources failing from specific regions

From Anomalies to Problems: Causal Analysis

When Davis detects anomalies across multiple entities around the same time, it doesn't fire separate alerts for each — it runs a causal analysis to determine which anomaly is the root cause and which are downstream effects.

The causal chain

Consider a realistic failure scenario:

1. Database host db-prod-02 starts experiencing disk I/O saturation
2. Query latency on the database increases
3. payment-service starts timing out on database calls
4. payment-service error rate spikes
5. checkout-service starts receiving errors from payment-service
6. checkout-service response time increases
7. User sessions start failing on the checkout flow

In a threshold-based system, every one of these events fires an alert. Davis sees the same events but traces the causal chain: db-prod-02 disk I/O saturation → database query latency → downstream service degradation. It creates a single Problem with db-prod-02 as the root cause.

What Davis uses for correlation

Davis pulls from multiple data sources to build its causal model:

• Smartscape topology — service dependency map; Davis knows checkout-service calls payment-service calls db-prod-02
• Event timeline — deployment events, config changes, resource saturation events
• PurePath traces — distributed traces that show the actual call chain
• Infrastructure metrics — host-level data correlated with service-level signals
• Synthetic monitor results — external availability data

When a deployment event lands at 14:32 and anomalies start appearing at 14:33, Davis flags the deployment as the likely trigger — even if the actual degradation takes minutes to propagate through the system.

Problem Cards: Reading the Output

A Problem card is Davis's output. It appears under Problems in the left nav and surfaces in the Home dashboard.

Anatomy of a Problem card

Header: Problem name (auto-generated), severity (Critical / High / Medium), status (Open / Resolved), duration.

Impact summary:

• Services affected: 3
• Users affected: 1,247
• Estimated revenue impact (if configured with user-value weighting)

Root cause section: The entity Davis identified as the source. Usually a specific host, process, service, or external dependency. Includes the specific metric that deviated and by how much.

Evidence: The data points supporting the root cause conclusion:

• Metric chart showing the anomaly onset
• Correlation score (Davis's confidence in the root cause hypothesis)
• Linked events (deployment, config change, infrastructure event)

Affected entities: The services and infrastructure components impacted, with their individual anomaly timelines.

Timeline: A chronological view of when each anomaly appeared and how the Problem evolved.

Problem severity levels

Severity is calculated based on the number of affected users, the business criticality of affected services (which you can configure via management zones), and the depth of the failure.

Configuring Anomaly Detection

Davis works out of the box, but you can tune it for your environment.

Global anomaly detection settings

Settings > Anomaly detection > Services:

• Response time: automatic (recommended) or manual threshold
• Failure rate: automatic or manual percentage threshold
• Throughput: automatic or manual drop threshold

For most services, automatic is correct. Set manual thresholds only when you have specific SLA commitments that differ from what Davis would naturally learn (e.g., a batch job where a 10-minute response time is normal but 30 minutes is an SLA breach).

Per-service overrides

Override detection sensitivity for individual services:

1. Navigate to the service: Services > [service name]
2. Settings (gear icon) > Anomaly detection
3. Set per-metric overrides

Use this when a service has inherently variable performance (a machine learning inference service with occasional long-running requests) that would otherwise generate noise.

Disabling detection for specific metrics

Sometimes a metric legitimately spikes in ways that aren't problems — a scheduled batch job that maxes out a queue every night. Suppress detection:

Settings > Anomaly detection > Infrastructure metrics > Add suppression rule

Define the entity, metric, and time window. Davis won't open Problems for that metric during the suppressed window.

Maintenance windows

For planned deployments, maintenance windows suppress all alerting:

Settings > Maintenance windows > Add maintenance window

Set the time range, affected management zones, and suppression type (alert only, or detect but don't alert). Davis still records events during maintenance windows — you can review them post-window.

Alert Settings and Notification Routing

Problems are Dynatrace's alerting unit. Configure where they go:

Built-in integrations

• PagerDuty: Settings > Integration > PagerDuty — maps Dynatrace severity to PagerDuty urgency
• Slack: Webhook-based. Set up a workflow that posts the Problem URL, affected services, and root cause summary
• OpsGenie: Native integration with alert deduplication
• ServiceNow: Create ITSM incidents automatically with entity metadata
• Email: Simple but consider whether it meets your response time SLA

Custom webhooks

For any system without a native integration, use the Problem notification webhook:

{
  "problemTitle": "{ProblemTitle}",
  "severity": "{ProblemSeverity}",
  "rootCauseEntity": "{RootCauseEntity}",
  "affectedServices": "{ImpactedEntities}",
  "problemUrl": "{ProblemURL}",
  "state": "{State}"
}

Dynatrace fires this payload when a Problem opens and again when it resolves (with "state": "RESOLVED").

Management zones for team routing

Management zones let you define subsets of your environment by team, application, or business unit. Combined with notification settings, you can route:

• Problems in the payments management zone → Payments Team PagerDuty
• Problems in the marketing management zone → Marketing Engineering Slack
• Problems in infrastructure → Platform Team OpsGenie

This eliminates the "everyone gets paged for everything" problem that plagues simpler APM setups.

Davis AI for Release Validation

One underused capability: using Davis as a release quality gate.

When you push a deployment, Davis watches for anomalies in the 60 minutes following the deployment event. If performance degrades post-deploy, Davis links the Problem to the deployment and surfaces it immediately.

Combine this with a deployment event push from your CI/CD pipeline:

# Push deployment event from CI
curl -X POST "https://<env>.live.dynatrace.com/api/v1/events" \
  -H "Authorization: Api-Token $DT_API_TOKEN" \
  -d '{
    "eventType": "CUSTOM_DEPLOYMENT",
    "deploymentName": "'"$SERVICE_NAME $VERSION"'",
    "deploymentVersion": "'"$VERSION"'",
    "entitySelector": "type(SERVICE),tag('"$SERVICE_NAME"')",
    "source": "GitHub Actions"
  }'

Then use the Events API to poll for Problems linked to the deployment entity in the post-deploy window. If a Problem opens with your deployment as the root cause trigger, fail the deployment pipeline.

This gives you observability-driven deployment validation — real production signals confirming the deploy is healthy, not just synthetic tests.

Grail and DQL: The Next-Gen Query Layer

Dynatrace is transitioning its backend to Grail (a unified data lakehouse) with DQL (Dynatrace Query Language) replacing older query interfaces. DQL is more powerful than USQL:

fetch logs
| filter service.name == "payment-service"
| filter log.level == "ERROR"
| summarize count(), by: bin(timestamp, 5m)
| sort timestamp desc

Davis's anomaly data is also queryable via DQL, enabling custom dashboards that surface Davis insights programmatically.

Where Davis Falls Short

Davis is excellent at correlating infrastructure and application signals. It has blind spots:

Business logic failures: If your order processing service silently accepts orders but writes them to the wrong database, response times look fine and error rates are zero. Davis won't catch it. You need functional testing for this — validating that the business outcome (order confirmed, inventory decremented) is correct, not just that the service responded.

Pre-production environments: Davis only monitors what's running. It can't tell you whether the code you're about to deploy will cause problems. For that, you need pre-deploy testing. HelpMeTest runs automated user journey tests against your staging environment — login flows, checkout paths, critical user actions — before the code ships. Davis then watches production after it deploys. The two systems are complementary: functional correctness before deploy, performance and availability observation after.

External user experience: Davis sees your infrastructure. It doesn't see a user in a slow network in Singapore trying to complete a purchase. Synthetic monitoring covers that gap — which is why Dynatrace Synthetic integrates with Davis to surface those failures as Problems.

Practical Tuning Tips

After running Dynatrace in production at scale, here are the tuning decisions that matter most:

Wait 2 weeks before trusting baselines. The first week has noisy anomaly detection. Week two, Davis has seen a full weekday/weekend cycle and the baselines stabilize.

Tag your entities from day one. Management zones, environment tags (environment:production, team:payments) make Davis's output actionable. An un-tagged Problem says "a process failed." A tagged Problem says "a payments team service in production failed."

Enable deployment events in your CI/CD. Davis's root cause analysis is significantly more accurate when it has deployment events to correlate with. Without them, it can identify the failing entity but can't confirm whether a code change triggered it.

Set critical service flags. Mark your core revenue-generating services as critical in Dynatrace. Davis weights impact severity by service criticality — a 100ms degradation on your checkout service is more critical than the same degradation on an internal admin panel.

Davis AI is not magic. It's a well-engineered system for correlating signals that humans are bad at correlating at 3am. Configure it well, give it context (tags, deployment events, management zones), and it becomes one of the most valuable tools in your reliability stack.