Dynatrace Kubernetes Observability: Full Setup with the Operator

Kubernetes adds a new layer of operational complexity to every application it runs. Pods restart. Nodes evict workloads. Resource limits throttle containers at unexpected times. Without proper observability, you're flying blind in a dynamic environment that changes constantly. Dynatrace Kubernetes observability gives you full-stack visibility — from cluster nodes down to individual traces inside pods — with minimal configuration. This guide walks through installation, what gets monitored automatically, and how to interpret the data.

Architecture: How Dynatrace Monitors Kubernetes

Dynatrace uses two components for Kubernetes monitoring:

OneAgent DaemonSet: A pod running on every node, responsible for instrumenting processes, collecting host metrics, and capturing container-level data.

Dynatrace Operator: A Kubernetes Operator that manages the lifecycle of Dynatrace components — deploying and updating OneAgent, configuring ActiveGate for cluster metrics, and handling webhook injection for auto-instrumentation.

The Operator is the recommended installation method. It handles upgrades, configuration changes, and cloud provider integration (EKS, GKE, AKS) automatically.

Installing the Dynatrace Operator

Prerequisites

• kubectl configured for your cluster
• Helm 3.x installed
• A Dynatrace environment (SaaS or Managed)
• An API token with the following scopes:
  • activeGateTokenManagement.read
  • entities.read
  • settings.write
  • settings.read
  • DataExport
  • InstallerDownload

Step 1: Add the Helm repository

helm repo add dynatrace https://raw.githubusercontent.com/Dynatrace/dynatrace-operator/main/config/helm/repos/stable
helm repo update

Step 2: Create the namespace

kubectl create namespace dynatrace

Step 3: Create the API token secret

kubectl create secret generic dynakube \
  --namespace dynatrace \
  --from-literal=apiToken=<YOUR_API_TOKEN> \
  --from-literal=dataIngestToken=<YOUR_DATA_INGEST_TOKEN>

Step 4: Install the Operator

helm install dynatrace-operator dynatrace/dynatrace-operator \
  --namespace dynatrace \
  --atomic

Verify the Operator is running:

kubectl get pods -n dynatrace
# NAME                                    READY   STATUS    RESTARTS   AGE
# dynatrace-operator-6d8b9f7c4b-x2pqk    1/1     Running   0          2m

Step 5: Create a DynaKube custom resource

The DynaKube resource configures what Dynatrace monitors. Create dynakube.yaml:

apiVersion: dynatrace.com/v1beta1
kind: DynaKube
metadata:
  name: dynakube
  namespace: dynatrace
spec:
  apiUrl: https://<YOUR-ENV>.live.dynatrace.com/api
  
  # OneAgent on every node
  oneAgent:
    classicFullStack:
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
          operator: Exists
  
  # ActiveGate for Kubernetes API integration
  activeGate:
    capabilities:
      - kubernetes-monitoring
      - routing
    resources:
      requests:
        cpu: 500m
        memory: 512Mi
      limits:
        cpu: 1000m
        memory: 1Gi
  
  # Kubernetes cluster monitoring
  metadataEnrichment:
    enabled: true
  
  namespaceSelector:
    matchLabels:
      dynatrace-monitoring: "true"

Apply it:

kubectl apply -f dynakube.yaml

The Operator will deploy OneAgent pods as a DaemonSet and spin up an ActiveGate deployment.

What Gets Monitored Automatically

Once the Operator is running, Dynatrace starts collecting:

Node-level metrics

For every node in the cluster:

• CPU utilization (system, user, iowait)
• Memory available, used, cached
• Disk throughput and IOPS
• Network bytes in/out, packet loss
• Node condition (Ready, MemoryPressure, DiskPressure, PIDPressure)

Pod and container metrics

For every pod:

• CPU requests vs limits vs actual usage
• Memory requests vs limits vs actual usage (RSS, working set)
• Container restart count
• Pod phase (Running, Pending, Failed, Succeeded)
• OOMKilled events

Workload health

Dynatrace tracks workload-level health for Deployments, StatefulSets, DaemonSets, and CronJobs:

• Desired vs ready replicas
• Failed pod percentage
• Rollout status

Kubernetes events

Critical cluster events are captured and correlated with metrics:

• Pod evictions (with reason: OOMKill, node pressure)
• Image pull failures
• Liveness probe failures
• Node drain/taint changes

Application Auto-Instrumentation

Auto-instrumentation injects the Dynatrace OneAgent SDK into pods without modifying application code or Docker images. It uses a Kubernetes admission webhook.

Enabling auto-instrumentation

Label the namespace you want instrumented:

kubectl label namespace production \
  instrumentation.opentelemetry.io/inject-java=true

Or use Dynatrace's webhook-based approach. Add the annotation to your deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
  namespace: production
spec:
  template:
    metadata:
      annotations:
        oneagent.dynatrace.com/inject: "true"
    spec:
      containers:
        - name: payment-service
          image: yourrepo/payment-service:v2.3.1

When the pod starts, the webhook injects the OneAgent init container and sets the required environment variables. Your application gets full distributed tracing, method-level profiling, and automatic exception capture — without changing a single line of application code.

Supported runtimes

For Go, use the Dynatrace Go Trace SDK explicitly in your code:

import dt "github.com/dynatrace/oneagent-go-sdk/api"

func handleRequest(w http.ResponseWriter, r *http.Request) {
    tracer := dt.CreateCustomTracer(nil)
    defer tracer.End()
    // ... your handler code
}

Namespace Monitoring and Filtering

Large clusters have dozens of namespaces. You probably don't want to instrument kube-system pods with full APM. Use namespaceSelector in your DynaKube to control which namespaces get instrumented:

spec:
  namespaceSelector:
    matchExpressions:
      - key: kubernetes.io/metadata.name
        operator: NotIn
        values:
          - kube-system
          - kube-public
          - cert-manager
          - ingress-nginx

This restricts OneAgent injection to application namespaces only, reducing resource overhead and noise in your Dynatrace environment.

Kubernetes Dashboards in Dynatrace

Dynatrace ships a dedicated Kubernetes view under Infrastructure > Kubernetes.

Cluster overview

The cluster overview shows:

• Total nodes and their health status
• Total pods (running / pending / failed)
• Namespace count
• CPU and memory utilization across the cluster
• Recent events (evictions, probe failures, OOMKills)

Namespace drill-down

Click a namespace to see:

• All workloads in the namespace
• CPU and memory usage per workload
• Pod restart count and health
• Recent events scoped to the namespace

Pod-level detail

Click a pod to see:

• Container metrics (CPU, memory per container)
• Environment variables (non-secret ones)
• Labels and annotations
• Log output (if log ingestion is configured)
• PurePaths originating from this pod (click-through to distributed traces)

The connection from pod → trace → service is what makes Dynatrace's Kubernetes monitoring more than a metrics dashboard. You can start from "this pod is consuming too much CPU" and click directly into the traces that show which requests are driving that CPU.

Resource Limits and Throttling Detection

One of the most common Kubernetes performance problems is CPU throttling — a container hitting its CPU limit and getting throttled by the kernel's CFS scheduler. This shows up as increased latency that's hard to attribute without the right metrics.

Dynatrace captures container_cpu_cfs_throttled_seconds_total and surfaces it in the pod view. When a container is being throttled, you'll see it flagged in the container metrics with the percentage of time the container was throttled.

Use this to identify containers that need their CPU limits raised or whose code is CPU-inefficient.

Resource right-sizing

Dynatrace's Resource Optimization (under Kubernetes settings) analyzes actual usage vs requests/limits and suggests adjustments:

• Containers requesting 2 CPU but averaging 0.1 — recommend reducing requests to 0.2
• Containers regularly hitting memory limits and OOMKilling — recommend increasing limits

This feeds into cost optimization work, particularly important in cloud environments where over-provisioned clusters waste budget.

Alerting on Kubernetes Health

Davis AI integrates Kubernetes events into its causal analysis. A pod restart storm caused by a bad deployment shows up as a Problem, not 50 separate alerts.

Configure specific Kubernetes anomaly detection under Settings > Anomaly detection > Kubernetes:

• Pod crash looping: Alert when a pod restarts more than N times in M minutes
• Not enough CPU: Alert when CPU requests can't be scheduled
• Memory limits: Alert when containers are approaching memory limits
• Pending pods: Alert when pods remain in Pending state beyond a threshold

Recommended alert thresholds

Integrating with CI/CD for Deployment Validation

When you deploy to Kubernetes via your CI/CD pipeline, push a deployment event to Dynatrace:

# GitHub Actions example
- name: Notify Dynatrace of deployment
  run: |
    curl -X POST \
      "https://$DT_ENV.live.dynatrace.com/api/v1/events" \
      -H "Authorization: Api-Token $DT_API_TOKEN" \
      -H "Content-Type: application/json" \
      -d "{
        \"eventType\": \"CUSTOM_DEPLOYMENT\",
        \"deploymentName\": \"$SERVICE_NAME $IMAGE_TAG\",
        \"deploymentVersion\": \"$IMAGE_TAG\",
        \"entitySelector\": \"type(SERVICE),tag($SERVICE_NAME),tag(environment:production)\",
        \"source\": \"GitHub Actions\",
        \"deploymentProject\": \"$GITHUB_REPOSITORY\"
      }"

Davis then correlates any post-deploy anomalies with this event, dramatically improving root cause accuracy.

Combining Dynatrace with Pre-Deploy Testing

Dynatrace Kubernetes observability is excellent at detecting problems in running clusters. It does not validate that new code is correct before it runs.

A solid Kubernetes deployment workflow looks like this:

1. PR opened → run unit tests, integration tests
2. PR merged → deploy to staging namespace
3. Staging deploy → run end-to-end user journey tests against staging (this is where HelpMeTest fits — automated tests for login flows, checkout, critical user paths)
4. Tests pass → deploy to production namespace
5. Production deploy → push deployment event to Dynatrace; Davis monitors for post-deploy anomalies

Dynatrace handles steps 4-5. The gap is step 3 — validating that the application actually works correctly before it ships. That's functional testing territory, and it's a different concern from observability.

Multi-Cluster and Hybrid Environments

If you run multiple clusters (production, staging, regional clusters), each cluster needs its own DynaKube resource connected to the same Dynatrace environment. Use management zones to separate cluster data:

metadata:
  annotations:
    dynatrace.com/management-zone: "production-us-east-1"

This lets your ops team filter to specific clusters while keeping all data in one Dynatrace environment for cross-cluster correlation.

For hybrid environments with VMs alongside Kubernetes — common during Kubernetes migrations — OneAgent on VMs and OneAgent in pods both report to the same Dynatrace environment. Smartscape shows the full topology: a Kubernetes service calling an on-VM database, with the dependency line drawn automatically.

Troubleshooting Common Installation Issues

OneAgent pods in CrashLoopBackOff: Check API token permissions. The token needs InstallerDownload scope. Run kubectl logs -n dynatrace <oneagent-pod> and look for 403 Forbidden responses.

Auto-instrumentation not working: Verify the admission webhook is deployed: kubectl get mutatingwebhookconfigurations | grep dynatrace. If missing, the Operator may have failed to register it. Check kubectl logs -n dynatrace deployment/dynatrace-operator.

Metrics missing for a namespace: Check the namespaceSelector in your DynaKube. If you're using label selectors, ensure the namespace has the required label: kubectl label namespace my-app dynatrace-monitoring=true.

High resource usage by OneAgent: OneAgent is a host-level agent and has real overhead. Typical: 1-2% CPU, 200-400MB memory per node. If you see significantly more, check if heap analysis is enabled — it's expensive. Disable it under Settings > OneAgent features if you don't need it.

Getting Dynatrace running on Kubernetes takes about 30-60 minutes. The return is full-stack visibility: cluster health, workload performance, distributed traces, and AI-driven root cause analysis — all correlated automatically. That's worth the setup time.