async-profiler and YourKit: JVM Performance Profiling in Production

Most Java performance problems fall into three categories: CPU hotspots (a method consuming disproportionate CPU time), memory leaks (heap growing without bound), and lock contention (threads waiting for synchronized resources). Identifying which category you're in—and which specific code is responsible—requires a profiler. async-profiler is the open-source choice for production profiling; YourKit adds GUI and deeper heap analysis.

async-profiler: Low-Overhead Production Profiling

Why async-profiler?

Traditional JVMTI-based profilers (JProfiler, older YourKit) use a safepoint mechanism that introduces bias: only sample at GC safepoints, missing code between them. async-profiler uses AsyncGetCallTrace (an internal JVM API) to sample at arbitrary points, giving accurate flame graphs even for CPU-intensive loops.

Typical overhead: 1-5% CPU, making it safe to run in production.

Installation

# Download latest release
wget https://github.com/async-profiler/async-profiler/releases/download/v3.0/async-profiler-3.0-linux-x64.tar.gz
tar xzf async-profiler-3.0-linux-x64.tar.gz

# Or macOS
wget https://github.com/async-profiler/async-profiler/releases/download/v3.0/async-profiler-3.0-macos.zip
unzip async-profiler-3.0-macos.zip

Attaching to a Running JVM

# Find the JVM process
jps -l
# Output: 12345 com.example.MyApp

# Profile for 30 seconds and generate flame graph
./profiler.sh -d 30 -f profile.html 12345

# CPU profiling (default)
./profiler.sh start -e cpu 12345
./profiler.sh stop -f profile.html 12345

# Wall-clock profiling (includes I/O wait time)
./profiler.sh -d 30 -e wall -f profile.html 12345

# Allocation profiling (find where objects are created)
./profiler.sh -d 30 -e alloc -f profile.html 12345

Starting with the JVM Agent

Add to JVM startup flags for always-on profiling:

java -agentpath:/path/to/libasyncProfiler.so=start,file=/tmp/profile.html,interval=1ms \
     -jar myapp.jar

Or in Docker:

ENV JAVA_OPTS="-agentpath:/opt/profiler/libasyncProfiler.so=start,file=/tmp/profile-%t.html,interval=1ms"

JFR (Java Flight Recorder) Output

async-profiler can write JFR format, readable by Java Mission Control:

./profiler.sh -d 60 -o jfr -f recording.jfr 12345

# Open in JMC
jmc recording.jfr

JFR format includes more metadata than HTML flame graphs: thread states, GC events, JIT compilation, and class loading.

Reading Flame Graphs

A flame graph shows the call stack of all samples taken during profiling:

• Y-axis: Call stack depth (bottom = thread entry point, top = actual executing code)
• X-axis: Cumulative time (width proportional to samples)
• Color: Random—color has no meaning in standard flame graphs

Reading technique:

1. Look for wide bars at the top of stacks — these are the actual hotspots
2. A wide bar means the method was at the top of the stack in many samples = consuming CPU
3. Follow the call chain downward to understand what triggered the hotspot

[mainThread]
  ├── processOrders()          width=100% (all time spent in this method's subtree)
  │   ├── validateOrder()      width=40%
  │   │   └── regexMatch()     width=35%  ← HOTSPOT: regex is slow
  │   └── persistOrder()       width=55%
  │       ├── serialize()      width=10%
  │       └── executeQuery()   width=45%  ← HOTSPOT: database calls

regexMatch is 35% of CPU time — consider pre-compiling the Pattern instead of compiling on every call.

Common async-profiler Findings

CPU Hotspot: Regex Compilation

// BAD: Pattern compiled on every call
public boolean isValidEmail(String email) {
    return email.matches("[a-z]+@[a-z]+\\.[a-z]+");
}

// GOOD: Compiled once
private static final Pattern EMAIL = Pattern.compile("[a-z]+@[a-z]+\\.[a-z]+");

public boolean isValidEmail(String email) {
    return EMAIL.matcher(email).matches();
}

The flame graph shows Pattern.compile() as the hotspot — a one-line fix eliminates 35% CPU usage.

Memory Allocation Hotspot

# Allocation profiler shows which methods allocate most
./profiler.sh -d 30 -e alloc -f alloc.html 12345

Common finding: String.format() inside loops creating thousands of temporary String objects. Replace with StringBuilder or String.concat().

Lock Contention

# Lock contention profiling
./profiler.sh -d 30 -e lock -f locks.html 12345

Shows which synchronization points have threads waiting. If 40% of samples show threads blocked on synchronized (cache), the cache access pattern needs redesigning (use ConcurrentHashMap instead of synchronized HashMap).

YourKit Java Profiler

YourKit is commercial ($499+) but offers capabilities async-profiler doesn't:

• GUI with IDE integration (IntelliJ IDEA, Eclipse)
• Heap snapshot analysis with retained size and object retention trees
• Automatic leak detection comparing snapshots
• Thread timeline visualization
• SQL query tracking with JDBC integration

Attaching YourKit

# Start JVM with YourKit agent
java -agentlib:yjpagent -jar myapp.jar

# Or attach to running process
# Open YourKit GUI → Connect → enter PID

Memory Leak Detection with YourKit

1. Take a heap snapshot before suspected leak
2. Run workload (process 1000 requests)
3. Force GC: System.gc() or from YourKit UI
4. Take second heap snapshot
5. Compare: YourKit → Comparison → Objects created since snapshot 1

The comparison table shows which classes grew. If byte[] grew by 500MB, drill down to see which objects reference them. The retention tree shows the GC root chain preventing garbage collection.

Continuous Profiling with Pyroscope

For always-on production profiling without manual attachment:

# Install Pyroscope server
docker run -d --name pyroscope \
  -p 4040:4040 \
  grafana/pyroscope

# Java agent
java -javaagent:pyroscope.jar \
  -Dpyroscope.application.name=myapp \
  -Dpyroscope.server.address=http://localhost:4040 \
  -jar myapp.jar

Pyroscope continuously samples your application and stores flame graphs over time. You can query: "What was CPU usage at 3:42 PM when the incident occurred?" and see the exact flame graph from that moment.

Profiling Spring Boot Applications

Spring Boot apps have common profiling targets:

Jackson Serialization

If /api/products is slow and the flame graph shows wide bars in ObjectMapper.writeValue(), you're serializing too many fields or using expensive types (LocalDateTime without Jackson module).

// Add Jackson modules for faster date serialization
@Configuration
public class JacksonConfig {
    @Bean
    public Jackson2ObjectMapperBuilderCustomizer jsonCustomizer() {
        return builder -> builder.modules(new JavaTimeModule());
    }
}

N+1 Query Detection

Flame graph shows many calls to HibernateJdbcCoordinator.executeQuery(). This indicates N+1: loading 100 orders, then loading each order's user separately (101 queries instead of 1 with JOIN).

Enable Hibernate statistics to count queries:

spring.jpa.properties.hibernate.generate_statistics=true
logging.level.org.hibernate.stat=DEBUG

Output: HHH000117: HQL: select order from Order, time: 2ms, rows: 100 followed by 100 individual user queries.

Fix with @EntityGraph or JOIN FETCH:

@Query("SELECT o FROM Order o JOIN FETCH o.user WHERE o.status = :status")
List<Order> findByStatus(String status);

Profiling Kotlin and Scala Applications

async-profiler works transparently for any JVM language. Kotlin and Scala compile to JVM bytecode, but frame names differ:

Kotlin coroutines show as $invokeSuspend frames in flame graphs. To see the actual coroutine logic, use the -e cpu mode and look for the underlying function calls below the suspend frame:

# Kotlin coroutine profiling — wall-clock shows all suspend points
./profiler.sh -d 30 -e wall -f kotlin-profile.html $(pgrep java)

If 60% of wall time is in DefaultDispatcher-worker-1 threads all showing park(), you have coroutines blocked waiting for something—usually IO or a mutex. Switch to alloc mode to find where allocations concentrate.

Scala: case class creation in hot paths appears as many new calls to generated classes. If Product.apply() (from case class companion) is wide in the flame graph, the path creates too many short-lived objects. Consider using mutable builders or primitive arrays in tight loops.

JVM Tuning Flags That Affect Profiling

Some JVM flags interfere with profiling or change what you observe:

# Always add when profiling to preserve stack frames
-XX:+PreserveFramePointer

# Disable JIT inlining to see individual method boundaries
# (Warning: significantly slows the JVM, use only to confirm findings)
-XX:-Inline

# Increase JIT compilation aggressiveness for more realistic hotspots
-XX:CompileThreshold=1000

# Enable tiered compilation (default in Java 8+, ensure it's on)
-XX:+TieredCompilation

-XX:+PreserveFramePointer is the most important: without it, frame pointer-based stack unwinding fails for some native frames, creating gaps in flame graphs. Always profile with this flag set.

Interpreting GC Overhead in Flame Graphs

GC activity appears in flame graphs as GarbageCollectionLocker or VMThread frames. If your flame graph shows 15-20% of CPU in GC-related frames:

1. Check heap size: jstat -gc $(pgrep java) 1000 10 — watch YGC (young GC) and FGC (full GC) counts
2. Profile allocations to find the producer: ./profiler.sh -d 30 -e alloc -f alloc.html $(pgrep java)
3. Common findings: creating Optional<> objects in tight loops, String concatenation with + in loops, boxing primitives in collections (Integer instead of int[])

GC pressure fix: use primitive collections (Eclipse Collections, Trove), pre-allocate arrays where size is known, and avoid creating temporary objects in hot paths.

Production Profiling Checklist

Before profiling in production:

1. Notify operations team: Profiling adds ~1-5% CPU overhead
2. Profile specific instances: Don't profile the entire cluster simultaneously
3. Time limit: Set a maximum profile duration (30-60s for CPU, 5 minutes max for allocation)
4. Store profiles: Write to a volume, not ephemeral container storage
5. Correlate with metrics: Match the profile timestamp to APM data showing the slowdown

# Profile a specific Kubernetes pod
kubectl exec -it my-pod-abc123 -- \
  /opt/profiler/profiler.sh -d 60 -f /tmp/profile.html $(pgrep java)

kubectl cp my-pod-abc123:/tmp/profile.html ./profile-$(date +%Y%m%d-%H%M).html

Summary

async-profiler provides accurate, low-overhead CPU and allocation profiling for JVM applications in production. Attach it to a running process with profiler.sh, profile for 30-60 seconds, and read the resulting flame graph: wide bars at the top of call stacks are your hotspots. YourKit adds heap comparison for memory leak analysis when async-profiler's allocation data isn't enough. For long-term visibility, continuous profiling with Pyroscope stores flame graphs over time so you can diagnose incidents after the fact. The pattern: profile, find the widest bar, fix it, verify the flame graph changed.