Flame Graphs and Continuous Profiling: Visualizing Performance at Scale

Flame graphs were invented by Brendan Gregg at Netflix to make profiling data human-readable. Instead of a table of function names and call counts, a flame graph shows the entire call stack hierarchy in a single visualization. The wide bars that stand out at the top of the graph are where your application spends its time—and what to fix first.

Anatomy of a Flame Graph

A flame graph has three properties:

• Each bar is a stack frame: The function name is shown inside the bar
• Y-axis (height) = stack depth: The bottom bar is the entry point (main thread, HTTP handler); bars above it are the functions that were called
• X-axis (width) = cumulative sample time: Wider bars were seen in more samples = more CPU time consumed

                                            ┌─────────────────────────────────┐
                                            │ hashCode() [3%]                 │  ← leaf (hotspot)
                              ┌─────────────┴─────────────────────────────────┴────────┐
                              │ equals() [20%]                                           │
              ┌───────────────┴──────────────────────────────────────────────────────────┤
              │ HashMap.get() [45%]                                                       │
┌─────────────┴───────────────────────────────────────────────────────────────────────────┴──┐
│ processRequest() [100%]                                                                      │
└──────────────────────────────────────────────────────────────────────────────────────────────┘
[thread: http-nio-8080-exec-1]

In this example: processRequest() is on the stack 100% of the time (it's the HTTP handler). It calls HashMap.get() 45% of the time, which calls equals(), which calls hashCode(). The hashCode() appearing at the top with 3% width is the leaf node—that's actually executing. Fix hashCode() implementation or reduce map lookups.

Reading Flame Graphs: What to Look For

Wide Bars at the Top

Wide bars at the top of their subtree (leaf nodes or near-leaf) are hotspots—the code actually consuming CPU.

┌─────────────────────────────────────────────────────────┐
│ serializeToJson() [60%]                                  │   ← Wide at top = HOTSPOT
├──────────────┬───────────────────────────────────────────┤
│ toJsonString │ ObjectMapper.writeValue() [55%]            │
│ [5%]         └─────────────────────┬─────────────────────┘
│               Jackson internals ... │

ObjectMapper.writeValue() is 55% of CPU time. The solution: check if you're creating a new ObjectMapper on every request (expensive), whether your objects use reflection-heavy serialization, or if you need faster serialization (e.g., Jackson AfterBurner module).

Flat Stacks (Wide Bottom Bars)

If the bottom bar (your entry point) is wide but everything above is thin and varied, your code calls many different things without any single hotspot. This is healthy—performance is spread evenly.

Towers (Tall, Thin Spikes)

Deep recursion or long call chains with small widths. Usually not a performance problem unless the recursion itself is the issue.

Idle Time (In Wall-Clock Profiles)

Wall-clock profiling (not CPU profiling) shows time including I/O wait. You'll see frames like epoll_wait, read, SocketInputStream.read at the top. These aren't CPU hotspots—they're the application waiting for network or disk. If 70% of wall time is in I/O, you have a different problem than a CPU optimization task.

Differential Flame Graphs

Differential (diff) flame graphs compare two profiles: before and after a change. Blue = decreased, red = increased.

# Generate with async-profiler
./profiler.sh -d 30 -f before.html 12345
# Deploy new code
./profiler.sh -d 30 -f after.html 12345

# Generate diff (using Brendan Gregg's tools)
./flamegraph.pl --title "Diff (red=regression, blue=improvement)" \
  <(./stackcollapse.pl before.txt) \
  <(./stackcollapse.pl after.txt) > diff.svg

Reading diffs:

• Red bars: More CPU time after the change (regression)
• Blue bars: Less CPU time after the change (improvement)
• Unchanged: Same width in both profiles

If validateRequest() turns red after a PR, that PR introduced a regression in validation performance.

Continuous Profiling with Pyroscope

Continuous profiling runs sampling constantly in production, storing profiles over time. When an incident occurs, you query the profile for that exact time window.

Pyroscope Server Setup

# Docker Compose
version: '3.8'
services:
  pyroscope:
    image: grafana/pyroscope:latest
    ports:
      - "4040:4040"
    volumes:
      - pyroscope-data:/data

volumes:
  pyroscope-data:

Access UI at http://localhost:4040.

Java Application Integration

<!-- pom.xml -->
<dependency>
    <groupId>io.pyroscope</groupId>
    <artifactId>agent</artifactId>
    <version>0.14.0</version>
</dependency>

// Application startup
PyroscopeAgent.start(
    new Config.Builder()
        .setApplicationName("my-service")
        .setProfilingEvent(EventType.ITIMER)  // CPU profiling
        .setServerAddress("http://pyroscope:4040")
        .setLabels(Map.of(
            "environment", "production",
            "version", System.getenv("APP_VERSION"),
            "region", System.getenv("AWS_REGION")
        ))
        .build()
);

Or via Java agent:

java -javaagent:/opt/pyroscope.jar \
     -Dpyroscope.application.name=my-service \
     -Dpyroscope.server.address=http://pyroscope:4040 \
     -Dpyroscope.profiling.interval=10ms \
     -jar myapp.jar

Python Integration

pip install pyroscope-io

import pyroscope

pyroscope.configure(
    application_name="my-python-service",
    server_address="http://pyroscope:4040",
    auth_token=os.environ.get("PYROSCOPE_TOKEN", ""),
    tags={
        "environment": "production",
        "region": "us-east-1"
    }
)

Go Integration

import "github.com/grafana/pyroscope-go"

pyroscope.Start(pyroscope.Config{
    ApplicationName: "my-go-service",
    ServerAddress:   "http://pyroscope:4040",
    Logger:          pyroscope.StandardLogger,
    Tags:            map[string]string{"environment": "production"},
    ProfileTypes: []pyroscope.ProfileType{
        pyroscope.ProfileCPU,
        pyroscope.ProfileAllocObjects,
        pyroscope.ProfileAllocSpace,
    },
})

Querying Historical Profiles

When an incident occurs at 3:42 PM:

1. Open Pyroscope UI at http://pyroscope:4040
2. Select the service: my-service
3. Set time range: 3:30 PM - 4:00 PM
4. Select profile type: cpu or wall
5. View the flame graph for that incident window

You see exactly what the application was doing when response times spiked—no need to reproduce the incident.

Tag-Based Queries

Pyroscope supports filtering by tags. If you tagged profiles with region:

{app="my-service", region="us-east-1"}  vs  {app="my-service", region="eu-west-1"}

Compare regional performance to diagnose regional slowdowns.

Parca: CNCF Continuous Profiling

Parca is the CNCF alternative to Pyroscope, with Kubernetes-native deployment:

# parca.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: parca
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: parca
          image: ghcr.io/parca-dev/parca:latest
          args:
            - /parca
            - --config-path=/etc/parca/parca.yaml
          ports:
            - containerPort: 7070

# parca-config.yaml
object_storage:
  bucket:
    type: FILESYSTEM
    config:
      directory: /data

scrape_configs:
  - job_name: kubernetes-pods
    scrape_interval: 10s
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_parca_io_scrape]
        regex: "true"
        action: keep

Annotate pods for automatic scraping:

# In your Deployment spec
metadata:
  annotations:
    parca.io/scrape: "true"
    parca.io/port: "7071"

Correlating Profiles with APM

The most powerful workflow: correlate flame graphs with APM metrics (response time, error rate):

1. APM shows spike at 14:32:00 (response time p99 jumps from 200ms to 2000ms)
2. Pyroscope query for 14:30:00 - 14:35:00
3. Flame graph shows DatabaseConnectionPool.getConnection() is 70% of CPU — connection pool exhausted
4. Fix: Increase connection pool size or reduce connection hold time

Without continuous profiling, step 3 requires reproducing the incident—which is often impossible for production spikes.

Grafana Integration

Add Pyroscope as a data source in Grafana:

1. Grafana → Data Sources → Add → Grafana Pyroscope
2. URL: http://pyroscope:4040
3. Create a panel with the Flame Graph visualization type

Use Grafana's "Explore" to query Pyroscope alongside Prometheus metrics in the same time window. The split-panel view—metrics on top, flame graph on bottom—is the standard continuous profiling workflow.

Production Profiling Overhead

Continuous profiling sample rates and overhead:

Default Pyroscope uses 10ms (100 samples/second per thread). For production, 100ms is safe for most services.

eBPF-Based Profiling (No Agent)

Modern continuous profiling tools use eBPF to profile any process without code changes:

# Grafana Alloy with eBPF profiling
docker run -d \
  --pid=host \
  --privileged \
  -v /sys:/sys \
  -v /proc:/proc \
  grafana/alloy \
  run /etc/alloy/config.alloy

# config.alloy
pyroscope.ebpf "local" {
  forward_to = [pyroscope.write.default.receiver]
}

pyroscope.write "default" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

eBPF profiling works for any language—Go, Python, Java, Rust, C++—without agents. It profiles at the OS level, capturing CPU usage from the kernel's perspective.

Icicle Charts and Off-CPU Flame Graphs

Standard flame graphs show CPU time. Two variants extend this:

Icicle charts are flame graphs flipped upside down—roots at the top, leaves pointing down. Some tools (Speedscope, some browser DevTools views) default to icicle layout. The interpretation is identical: wider = more time, leaf nodes at the bottom are hotspots.

Off-CPU flame graphs show time spent NOT executing: waiting for I/O, locks, or being descheduled. Generate with async-profiler in wall-clock mode:

./profiler.sh -d 30 -e wall -f offcpu.html 12345

Reading off-CPU graphs:

• Wide futex, epoll_wait, or read at the top = I/O wait (healthy for I/O-bound services)
• Wide park or LockSupport.park = thread blocked on a lock (contention problem)
• Wide Thread.sleep = explicit sleeping (expected in scheduled tasks)

If 70% of off-CPU time is in Object.wait() leading back to a single synchronized method, that's your contention hotspot.

Flame Graph Anti-Patterns

Common mistakes when reading flame graphs:

Focusing on the widest bar at the bottom: The bottom bar is usually your entry point (main, HTTP handler) and is always wide because everything runs under it. Only leaf nodes and near-leaf nodes are actionable.

Ignoring thin call chains that sum to large width: A function that calls 50 different methods each taking 2% could represent 100% of a performance budget. Look for "busy" nodes with many diverse children.

Profiling under too little load: A flame graph at 10% CPU shows idle time, GC, and housekeeping. Profile under realistic load—use your actual request rate or run a load test simultaneously.

Trusting one sample: A 5-second profile can show noise. Run three 30-second profiles and compare. If the same hotspot appears in all three, it's real. If it varies, it's sampling artifact.

Sharing and Storing Flame Graphs

For team workflows, store flame graphs as artifacts alongside incidents:

# Name with context
cp profile.html "profiles/$(date +%Y%m%d)-$(git rev-parse --short HEAD)-incident-203.html"
git add profiles/
git commit -m "Add profile from incident-203 at 2026-05-28 14:32"

Or upload to Speedscope for sharing without requiring file downloads—paste the JSON at speedscope.app for interactive exploration and sharing via URL.

For CI, attach flame graphs as build artifacts:

# GitHub Actions
- name: Run load test with profiling
  run: |
    ./profiler.sh -d 60 -f profile-${{ github.sha }}.html $(pgrep java)

- name: Upload profile
  uses: actions/upload-artifact@v3
  with:
    name: cpu-profile-${{ github.sha }}
    path: profile-*.html

This creates a historical record: when did the flame graph change shape? PR #123 introduced the new serialization—and that's when the ObjectMapper bar got wide.

Summary

Flame graphs translate profiling samples into an actionable visual: find the widest bar at the top of its subtree, fix that function, repeat. Differential flame graphs make before/after comparisons objective—red regression, blue improvement. Continuous profiling with Pyroscope or Parca captures this data always-on in production, enabling post-hoc incident investigation without reproduction. The investment: add a Pyroscope agent to your services (5 lines of code per service), point it at a Pyroscope server, and gain permanent retrospective visibility into every performance spike you'll ever have.