Monitoring

Elastic APM: Performance Monitoring Within the ELK Stack

HelpMeTest

HelpMeTest

— 9 min read
Elastic APM: Performance Monitoring Within the ELK Stack

If your organization already runs the ELK stack — Elasticsearch, Logstash, Kibana — for log aggregation and search, Elastic APM extends that investment into application performance monitoring without introducing a new vendor or a new data platform. Traces, metrics, and logs flow into the same Elasticsearch cluster, become queryable through the same Kibana interface, and correlate against each other using shared identifiers.

This guide covers how Elastic APM works across different language runtimes, how to correlate APM data with logs, how to use it for performance testing workflows, and how it compares to commercial APM tools.

Elastic APM Architecture Overview

Elastic APM consists of three components:

APM Agents: Language-specific libraries that instrument your application code, capturing traces, errors, and metrics. Agents are available for Node.js, Python, Java, Go, Ruby, .NET, PHP, and browser JavaScript (Real User Monitoring). Each agent uses language-appropriate mechanisms — bytecode instrumentation for Java, monkey-patching for Python, middleware integration for Node.js.

APM Server: A standalone process (or Elastic Agent integration) that receives data from APM agents, processes it, and stores it in Elasticsearch. The APM Server speaks a binary protocol with agents and exposes a REST API for data ingestion.

Kibana APM UI: Pre-built dashboards and views in Kibana for exploring APM data: service maps, transaction traces, error groupings, and performance metrics.

Together, these components provide a complete distributed tracing and application monitoring solution that lives within your existing ELK infrastructure.

Agent Setup by Language

Node.js

The Elastic APM Node.js agent integrates as the first import in your application entry point:

// Must be the first import
const apm = require('elastic-apm-node').start({
  serviceName: 'checkout-service',
  secretToken: process.env.ELASTIC_APM_SECRET_TOKEN,
  serverUrl: process.env.ELASTIC_APM_SERVER_URL,
  environment: process.env.NODE_ENV,
  // Instrument specific frameworks
  captureBody: 'errors',  // capture request bodies on errors
  captureErrorLogStackTraces: 'always'
});

The agent automatically instruments Express, Koa, Fastify, HTTP module calls, database clients (pg, mysql2, mongoose, redis, elasticsearch), and several message queue libraries. Manual instrumentation for custom operations:

const span = apm.startSpan('calculate-tax');
try {
  const result = await calculateTax(order);
  span.setLabel('order.total', order.total);
  return result;
} finally {
  span.end();
}

Python

The Python agent follows a similar pattern, with auto-instrumentation for Django, Flask, FastAPI, SQLAlchemy, Redis, and several HTTP client libraries:

import elasticapm
from elasticapm.contrib.flask import ElasticAPM

app = Flask(__name__)
app.config['ELASTIC_APM'] = {
    'SERVICE_NAME': 'inventory-service',
    'SECRET_TOKEN': os.environ.get('ELASTIC_APM_SECRET_TOKEN'),
    'SERVER_URL': os.environ.get('ELASTIC_APM_SERVER_URL'),
    'ENVIRONMENT': os.environ.get('FLASK_ENV', 'production'),
    'CAPTURE_BODY': 'errors',
}
apm = ElasticAPM(app)

For manual spans in Python:

with elasticapm.capture_span('external-pricing-api'):
    response = requests.get('https://pricing.internal/calculate', params=order_params)

Java

The Java agent uses Java instrumentation (the -javaagent JVM flag) to apply bytecode transformations without code changes:

java -javaagent:/path/to/elastic-apm-agent.jar \
  -Delastic.apm.service_name=payment-service \
  -Delastic.apm.server_url=http://apm-server:8200 \
  -Delastic.apm.secret_token=${ELASTIC_APM_SECRET_TOKEN} \
  -Delastic.apm.environment=production \
  -jar payment-service.jar

The Java agent automatically instruments Spring MVC, Spring Boot, Jakarta EE, JAX-RS, JDBC (detecting the underlying driver), JMS, Kafka, Elasticsearch client, and many other frameworks. Because it uses bytecode instrumentation, no application code changes are required — it's purely a JVM configuration change.

Go

Go's static compilation means bytecode instrumentation isn't possible. The Elastic APM Go agent provides a wrapper-based approach:

import (
    "go.elastic.co/apm/v2"
    "go.elastic.co/apm/module/apmhttp/v2"
    "go.elastic.co/apm/module/apmgorm/v2"
    "gorm.io/gorm"
)

// Wrap HTTP server
handler := apmhttp.Wrap(router)

// Wrap GORM database client
db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{})
db.Use(apmgorm.Plugin())

// Manual transaction context (required for trace propagation in goroutines)
func processOrder(ctx context.Context, order Order) error {
    tx := apm.TransactionFromContext(ctx)
    span := tx.StartSpan("validate-inventory", "external", nil)
    defer span.End()
    // ...
}

Go requires more explicit instrumentation than Java or Python, but the tracing data is fully integrated with the rest of the Elastic stack.

Traces and Spans in Kibana

Once agents are sending data, Kibana's APM section provides several views:

Service Overview: All instrumented services, each showing request rate, latency (mean, P95, P99), and error rate. This is the starting point for identifying which service has degraded.

Transaction Groups: Within a service, HTTP endpoints or background jobs grouped by type. Each transaction group shows its performance distribution and links to representative traces.

Trace Explorer: A waterfall view of individual distributed traces, showing the complete call chain across services. Each span shows its start time, duration, and any associated database queries or external calls.

Error Tracking: Application exceptions grouped by type and message, with stack traces, occurrence counts, and the trace context in which each error occurred.

Service Map: An auto-generated topology map showing how services call each other, with each edge showing error rates and latency for that communication path.

For QA teams, the Service Map provides immediate context when investigating failures: if your test is calling Service A, and Service A depends on Services B and C, the map shows whether the problem originated in Service B or C before you have to examine individual traces.

Correlating APM Data with Logs

This is where Elastic APM's integration with the ELK stack becomes particularly powerful. Every APM transaction and span has a trace.id and transaction.id. The Elastic APM agents inject these identifiers into the structured logging output of your application — and if you're already using structured logging with the Elastic Common Schema (ECS), this correlation is automatic.

For Node.js with Winston:

const winston = require('winston');
const apm = require('elastic-apm-node');

const logger = winston.createLogger({
  format: winston.format.combine(
    winston.format.json(),
    winston.format((info) => {
      // Inject APM correlation IDs
      const ids = apm.currentTraceIds;
      return Object.assign(info, ids);
    })()
  ),
  transports: [new winston.transports.Console()]
});

With this configuration, every log message emitted during a request includes trace.id and transaction.id. In Kibana, when you're viewing a trace, you can click "View Logs" to see all log messages emitted during that specific trace — across all services that handled the request.

This removes one of the most painful aspects of distributed debugging: correlating log messages from multiple services. Instead of searching through log files with timestamp ranges and guessing which service logs correspond to which request, you follow a single trace ID across all services simultaneously.

The ECS Advantage

The Elastic Common Schema (ECS) defines a standard set of field names for common log properties — timestamps, service names, host information, error details. When your application logs use ECS field names, Kibana can automatically apply the right visualizations and correlations without custom field mapping.

For teams heavily invested in the ELK stack, standardizing on ECS across application logs and APM data creates a unified data model where searching across logs, traces, and metrics uses consistent field names. service.name, error.type, user.id mean the same thing in every data source.

Performance Testing Workflows with Elastic APM

Elastic APM integrates naturally with performance testing because all test-generated traffic appears in the same APM data as production traffic (if you're testing against a production-like environment) or a dedicated test environment.

Marking Test Runs

To separate test traffic from other traffic, use APM labels to tag transactions with test context:

// Node.js - add labels to current transaction
apm.setLabel('test.run_id', process.env.CI_BUILD_ID);
apm.setLabel('test.suite', 'checkout');
apm.setLabel('environment.name', 'test');

These labels become Elasticsearch fields that you can filter on in Kibana. During a test run, you can view APM data filtered to labels.test.run_id: <your-build-id> to see only the performance of test-generated traffic.

Querying APM Data Programmatically

Elastic APM stores all data in Elasticsearch, which means you can query it using the Elasticsearch API. This is the foundation for programmatic performance quality gates:

from elasticsearch import Elasticsearch
from datetime import datetime, timedelta

es = Elasticsearch(
    hosts=[os.environ['ELASTICSEARCH_URL']],
    api_key=os.environ['ELASTICSEARCH_API_KEY']
)

def get_p99_latency(service_name, transaction_name, minutes=10):
    """Query P99 latency for a specific transaction over the last N minutes."""
    result = es.search(
        index="apm-*-transaction-*",
        body={
            "query": {
                "bool": {
                    "filter": [
                        {"term": {"service.name": service_name}},
                        {"term": {"transaction.name": transaction_name}},
                        {"range": {
                            "@timestamp": {
                                "gte": f"now-{minutes}m"
                            }
                        }}
                    ]
                }
            },
            "aggs": {
                "p99": {
                    "percentiles": {
                        "field": "transaction.duration.us",
                        "percents": [99]
                    }
                }
            },
            "size": 0
        }
    )
    p99_us = result["aggregations"]["p99"]["values"]["99.0"]
    return p99_us / 1000  # convert to milliseconds

def check_performance_gate(service, transaction, threshold_ms):
    p99 = get_p99_latency(service, transaction)
    if p99 > threshold_ms:
        raise AssertionError(
            f"Performance gate failed: {transaction} P99 = {p99:.0f}ms "
            f"(threshold: {threshold_ms}ms)"
        )

This approach gives you flexible, programmable quality gates based on real application performance data, not just load test tool metrics.

Correlating Load Test Phases with APM Data

When running load tests with tools like Gatling, k6, or JMeter, correlate test phases with APM data using Kibana annotations:

# Create an annotation at load test start
curl -X POST "${KIBANA_URL}/api/annotations" \
  -H "kbn-xsrf: true" \
  -H "Authorization: ApiKey ${KIBANA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d "{
    \"annotation\": {
      \"type\": \"annotation\",
      \"dashboardId\": \"apm-service-overview\",
      \"time\": \"$(date -u +%s000)\",
      \"message\": \"Load test start: ${TEST_SCENARIO}\"
    }
  }"

Annotations appear as vertical markers on time-series charts in Kibana, making it visually obvious where load test phases begin and end relative to performance metrics.

Self-Hosted vs Elastic Cloud

One of the key decisions for Elastic APM is where to run it. Both options have legitimate use cases.

Self-Hosted Elastic Stack

Running your own Elasticsearch cluster, Logstash (or Beats), and Kibana gives you full control over data residency, retention policies, and infrastructure costs — at the cost of operational complexity.

When self-hosted makes sense:

  • Compliance requirements mandate on-premises data storage
  • Your organization already has experienced Elasticsearch operators
  • You're processing high data volumes where cloud pricing becomes prohibitive
  • You need complete control over indexing and retention policies

Operational considerations:

  • Elasticsearch requires careful JVM heap tuning (typically 50% of available RAM, max 32GB per node)
  • Cluster sizing for APM workloads depends heavily on data retention period and request volume
  • Rolling upgrades require careful planning as APM agents have version compatibility requirements
  • Hot-warm-cold architecture significantly reduces storage costs for high-volume deployments

Elastic Cloud

Elastic Cloud is the managed Elastic Stack offering, available on AWS, GCP, and Azure. It handles cluster provisioning, upgrades, backups, and scaling automatically.

When Elastic Cloud makes sense:

  • Your team lacks Elasticsearch expertise
  • You need to be operational quickly
  • Your data volume is moderate (cloud pricing is manageable)
  • You want Elastic's native integrations with cloud provider services

Cost model: Elastic Cloud pricing is based on RAM hours for compute and GB/month for storage. For APM specifically, the APM Server is typically a small component — the cost is primarily driven by Elasticsearch storage and compute for trace data.

Elastic APM vs Commercial APM Tools

How does Elastic APM compare to Datadog, Dynatrace, and AppDynamics?

Depth of Auto-Instrumentation

Commercial tools generally offer deeper auto-instrumentation with less configuration. Dynatrace's OneAgent and AppDynamics's agents apply instrumentation with minimal setup and automatically discover dependencies. Elastic APM's agents require explicit configuration and have narrower auto-instrumentation coverage.

For teams running standard frameworks (Spring Boot, Express, Django), this difference is minor — common frameworks are well-covered. For teams with custom frameworks or less common technology stacks, commercial tools may offer better out-of-the-box coverage.

Analytics and Intelligence

Datadog's anomaly detection, Dynatrace's Davis AI, and AppDynamics's baselines are significantly more sophisticated than Elastic APM's alerting. Elastic APM relies on Elasticsearch and Kibana's general analytics capabilities rather than purpose-built APM analytics.

For teams comfortable with Elasticsearch query construction, this flexibility is an asset. For teams that want insights surfaced automatically, commercial tools provide a better experience.

Cost at Scale

At high data volumes, self-hosted Elastic APM is significantly cheaper than commercial APM tools, which typically charge per host or per data volume. A large deployment with hundreds of services can cost 10x less with Elastic APM than with Datadog or Dynatrace.

Integration Depth

For teams already running the ELK stack for logs, Elastic APM's native log correlation is a meaningful advantage over commercial APM tools. Correlating Datadog traces with Elasticsearch logs requires custom configuration; Elastic APM traces and ELK logs correlate automatically.

Building an Observability Practice with Elastic APM

Elastic APM rewards teams that invest in good instrumentation practices. Here's a pragmatic path to building observability:

Week 1-2: Basic instrumentation Deploy agents to your highest-traffic services, accept the defaults, and start getting familiar with Kibana APM. Identify the five business-critical transactions you care most about.

Week 3-4: Log correlation Add ECS-compatible structured logging to the same services. Enable APM log correlation. Now your traces link to logs.

Month 2: Custom context Add business-level labels to transactions: user tier, feature flags, order type. This transforms traces from technical artifacts into business-relevant data.

Month 3: Quality gates Integrate Elasticsearch queries into your CI pipeline. Establish performance baselines. Fail builds when transactions exceed baseline P99 by more than 20%.

Ongoing: Alert tuning Configure Kibana alerting rules based on patterns you've observed. Start with error rate and P99 thresholds, then refine as you learn your application's normal behavior.

Extending with Functional Testing

Elastic APM tells you how your application performs — which operations are slow, which are failing, what the root cause is. It doesn't tell you whether your application produces the correct results.

HelpMeTest complements Elastic APM by validating functional correctness across all the user-facing flows that your APM data covers. Running HelpMeTest automated tests against an Elastic APM-instrumented application means every functional test generates trace data in Kibana. When tests fail, the APM traces provide immediate context: was the failure caused by a slow database query, a downstream service error, or an application logic issue? The combination of functional test results and APM trace data dramatically shortens the debug cycle.

Conclusion

Elastic APM is the natural choice for teams already invested in the ELK stack. The native integration between traces, metrics, and logs eliminates the data silo problem that plagues teams running separate logging and APM platforms. The self-hosted option provides cost-effective APM at scale. The open-source codebase means no vendor dependency for instrumentation.

The main tradeoff is operational complexity (for self-hosted deployments) and more limited automatic insights compared to commercial APM tools. Teams with Elasticsearch expertise and budget sensitivity will find Elastic APM a strong fit. Teams prioritizing ease of use and automatic intelligence may find Datadog or Dynatrace worth the additional cost.

Start with auto-instrumentation for your core services, add log correlation, and build from there. The investment compounds quickly as observability data becomes a standard part of how your team debugs, tests, and operates software.

Read more

Start now free