Honeycomb.io Observability: High-Cardinality Debugging and BubbleUp Analysis

Honeycomb.io Observability: High-Cardinality Debugging and BubbleUp Analysis

Most monitoring tools were designed for a world that no longer exists. In that world, applications ran on fixed infrastructure, users behaved predictably, and performance problems could be diagnosed by checking a few standard metrics — CPU, memory, response time, error rate. When something was wrong, you looked at your dashboards and found the red graph.

Honeycomb.io was designed for a different world. In modern distributed systems, the interesting questions are specific and contextual: Which users are experiencing slow checkout? Is it worse for users in a particular region? Does it correlate with specific payment providers? Is it happening more for users with large shopping carts? These questions cannot be answered by dashboards full of averaged metrics. They require the ability to slice and dice high-dimensional event data on demand.

This guide explains how Honeycomb's approach to observability works, when it outperforms traditional monitoring, and how to use it effectively for debugging and quality assurance.

High-Cardinality Observability: What It Means and Why It Matters

Cardinality refers to the number of unique values a field can take. Low-cardinality fields have few possible values: HTTP status codes (a handful), region (a few dozen), environment (staging, production, development). High-cardinality fields have many possible values: user IDs (millions), request IDs (billions), shopping cart contents (unbounded).

Traditional monitoring tools are built around low-cardinality data. Metrics systems like Prometheus aggregate measurements into counters, gauges, and histograms — which means you pre-define the dimensions you want to query and discard everything else. You can tell that p95 latency is 800ms, but you can't filter that measurement to a specific user or a specific database query unless you pre-defined that dimension at collection time.

Honeycomb stores events in their raw, unaggregated form. Each event is a structured JSON document — a trace span, a request log, a custom event — with as many fields as your application generates. Honeycomb's query engine lets you filter, group, and aggregate across any combination of these fields at query time, not at collection time.

This means you can answer questions you didn't think to ask when you instrumented your application. A week after deployment, you can discover that slow responses correlate with a specific customer_tier field you added to traces for a different purpose.

How Honeycomb Differs from Traditional Monitoring

The difference in approach produces very different workflows when debugging production issues.

Traditional monitoring workflow:

  1. Alert fires: "p95 latency exceeded threshold"
  2. Check dashboards: find elevated latency graph
  3. Check related dashboards: database response time? Queue depth?
  4. Form hypothesis: "Maybe it's the database"
  5. Check database metrics: yes, query time is elevated
  6. Still unclear: which query? Which table? For which users?
  7. Dig into logs: search through raw log output
  8. Eventually find root cause, possibly 30-60 minutes later

Honeycomb workflow:

  1. Alert fires: "p95 latency exceeded threshold"
  2. Open Honeycomb, filter to slow requests (duration > 1000ms)
  3. Group by db.table: immediately see that inventory table queries are slow
  4. Group by user.region: see that it's worse for EU users
  5. Drill into specific traces: see exact query plans
  6. Root cause in 5-10 minutes

The key difference is that Honeycomb lets you explore the data interactively, following the evidence wherever it leads, rather than checking a fixed set of pre-built dashboards that may or may not contain the dimension you need.

BubbleUp: Automated Anomaly Dimension Discovery

Manually grouping by different dimensions to find the one that explains an anomaly can itself be tedious if you have hundreds of fields in your events. Honeycomb's BubbleUp feature automates this process.

When you identify a set of slow or erroneous requests — by selecting them in a histogram or by running a query — BubbleUp automatically compares the properties of those requests against the baseline of all requests. It returns a ranked list of fields and values that are statistically over-represented in the anomalous set.

In practice, BubbleUp might tell you:

Slow requests (>2s) are over-represented in:user.account_type = enterprise (4.2x more likely)payment.provider = stripe_intl (3.8x more likely)feature_flag.new_checkout = true (3.1x more likely)

This immediately narrows your investigation to the feature flag, which was just rolled out to enterprise users who are more likely to use international payment processing. Without BubbleUp, finding this three-way correlation would have required a lucky guess about which dimensions to check first.

For QA teams, BubbleUp is particularly valuable during testing. When a test run produces unexpected failures or performance regressions, BubbleUp can identify which properties of the failing requests distinguish them from passing ones — often revealing patterns the test failure messages themselves don't make obvious.

Query-Driven Debugging vs Dashboards

Honeycomb has dashboards, but they're not where the interesting work happens. The core interface is a query builder that lets you construct arbitrary queries over your event data.

A Honeycomb query has the following structure:

  • Dataset: which event stream to query (you can have multiple: traces, frontend events, custom events)
  • Filter: which events to include (e.g., service.name = checkout AND status_code >= 500)
  • Breakdown: group results by one or more fields (e.g., error.type, user.plan)
  • Calculate: what to compute (COUNT, P50, P95, P99, HEATMAP, etc.)
  • Time range: over what window to query

Queries can be composed and linked. You can start with a broad query to identify that something is wrong, click into a specific value in the results to add it as a filter, and progressively narrow your investigation without losing context.

This composability is what makes Honeycomb feel different from dashboard-based tools. You're not navigating between dashboards — you're interrogating the data with a series of questions, each informed by the previous answer.

Saved Queries and Boards

While query-driven investigation is Honeycomb's strength, saved queries and boards serve an important role for recurring checks. QA teams can save queries like:

  • "Error rate by service, last 24h"
  • "P95 latency for checkout, broken down by payment provider"
  • "Test suite failures by test name, last 7 days"

Boards collect these saved queries into a shared view, giving teams a common reference point without the rigidity of pre-built dashboards.

OpenTelemetry Instrumentation with Honeycomb

Honeycomb is one of the most OpenTelemetry-native observability platforms available. OpenTelemetry is the CNCF project that provides vendor-neutral APIs and SDKs for generating distributed traces, metrics, and logs. Honeycomb fully supports OTLP (OpenTelemetry Protocol) ingestion, and the company actively contributes to OpenTelemetry development.

Setting up OpenTelemetry instrumentation for Honeycomb typically takes less than 30 minutes for a typical service. Here's an example for a Node.js application:

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: 'https://api.honeycomb.io/v1/traces',
    headers: {
      'x-honeycomb-team': process.env.HONEYCOMB_API_KEY,
    },
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

The getNodeAutoInstrumentations() package automatically instruments Express, HTTP calls, database clients (PostgreSQL, MySQL, MongoDB, Redis), and several other common libraries. You get distributed tracing across service boundaries immediately.

Adding Custom Attributes for Testing Context

Auto-instrumentation gives you infrastructure-level data. To make Honeycomb genuinely useful for debugging your specific application, add custom attributes to your spans:

const { trace } = require('@opentelemetry/api');

async function processOrder(order) {
  const span = trace.getActiveSpan();
  if (span) {
    span.setAttributes({
      'order.id': order.id,
      'order.total': order.total,
      'order.item_count': order.items.length,
      'order.customer_tier': order.customer.tier,
      'feature_flag.new_pricing': isEnabled('new_pricing', order.customer)
    });
  }
  // ... order processing logic
}

These attributes become queryable dimensions in Honeycomb. When an order fails, you can immediately group failures by order.customer_tier or filter to feature_flag.new_pricing = true to narrow the investigation.

For test instrumentation specifically, add test context as span attributes:

// In your test setup
span.setAttributes({
  'test.suite': 'checkout',
  'test.case': 'guest_checkout_with_promo',
  'test.environment': process.env.TEST_ENV,
  'test.run_id': process.env.CI_BUILD_ID
});

This lets you filter Honeycomb data to traces generated by specific tests, specific test runs, or specific test environments.

Team-Based Analysis and Collaboration

Honeycomb is designed for teams to use together, not just for individual debugging sessions. Several features support collaborative analysis:

Permalinks: Every Honeycomb query generates a permanent, shareable URL. When you find something interesting, you can share the exact query (with all its filters and breakdowns) with a teammate. They see exactly what you're seeing, not just a description of it.

Trace comments: In the trace view, team members can add comments to specific spans. This turns a trace into a collaborative investigation artifact — developers, QA engineers, and operations staff can all annotate what they found and what it means.

Team boards: Boards can be shared across teams, creating a common operational picture without requiring everyone to know how to construct queries from scratch.

History: Honeycomb maintains your query history, so you can return to a previous investigation path if a new approach doesn't pan out.

For QA teams, the permalink feature is particularly valuable for bug reports. Instead of trying to describe what you observed, you include a Honeycomb permalink that shows the exact behavior. Developers can immediately see the traces, filter further if needed, and reproduce the exact query that surfaced the issue.

How Honeycomb Differs from Traditional Monitoring Tools

It's worth being explicit about what Honeycomb is not designed for, because the differences affect how you should combine it with other tools.

Honeycomb is not a metrics platform: If you need infrastructure metrics — CPU, memory, disk I/O, network throughput — you'll want a dedicated metrics platform like Prometheus or Datadog infrastructure monitoring alongside Honeycomb. Honeycomb's strength is in high-dimensional event data, not in raw infrastructure timeseries metrics.

Honeycomb is not a log aggregation platform: While Honeycomb can ingest structured log events, it's not a replacement for Elasticsearch/OpenSearch or Splunk for full-text log search. Honeycomb excels at structured, schema-defined events with known field names, not at free-form text search.

Honeycomb requires instrumentation investment: Unlike Dynatrace or Instana, which auto-instrument most of your stack without code changes, Honeycomb requires you to explicitly instrument your application (either with OpenTelemetry or a Honeycomb SDK). This is a meaningful upfront investment, but it produces richer, more meaningful data because the instrumentation reflects your application's business logic rather than just its technical behavior.

The tradeoff: Honeycomb's data is harder to generate but much more powerful to query. Teams that invest in good instrumentation find it pays dividends in dramatically faster debugging.

Practical Observability-Driven Testing with Honeycomb

Here's how a QA team can incorporate Honeycomb into their testing workflow:

Instrument test scenarios: Add test context attributes to your spans so Honeycomb data is filterable by test case and test run. This is the foundation of everything else.

Build test-specific queries: Create saved queries for your most common test analysis needs: error rate during test runs, latency breakdown by test scenario, error type distribution.

Use BubbleUp for flaky test investigation: When tests fail intermittently, run BubbleUp on the failing spans to identify which properties distinguish failing requests from passing ones. Flaky tests often have environmental or data-dependent causes that BubbleUp will surface quickly.

Include Honeycomb links in CI reports: When your CI pipeline completes a test run, generate Honeycomb links for the relevant time window and include them in the test report. Engineers investigating failures have immediate access to trace data without manual searching.

Track test-induced regressions over time: Use Honeycomb's comparison feature to compare performance metrics between test run windows. A query showing P95 latency over the last 30 test runs will surface gradual regressions that single-run analysis misses.

Combining Honeycomb with Functional Testing

Honeycomb's observability capabilities tell you how your application behaves internally — which services are slow, which queries are expensive, which users are affected. What it doesn't do is verify that your application behaves correctly from the user's perspective.

HelpMeTest provides automated functional testing that validates the actual user experience across your application's critical flows. When you run HelpMeTest tests against a Honeycomb-instrumented application, every test interaction generates trace data you can explore in Honeycomb. Failures get the full benefit of Honeycomb's query-driven debugging — you can immediately identify whether a functional test failure is caused by a slow database, an upstream service error, or an application logic bug.

Conclusion

Honeycomb represents a fundamentally different approach to observability — one optimized for the questions you don't know to ask yet. Its high-cardinality data model, query-driven interface, and BubbleUp automation address the core challenge of debugging distributed systems: you don't know which dimension matters until you find it.

For QA teams, the investment in OpenTelemetry instrumentation with business-level context pays off in dramatically faster root cause analysis, better collaboration with developers, and the ability to detect patterns in test failures that simpler monitoring tools would miss. Start with auto-instrumentation to get traces flowing, add business-context attributes to your spans, and build the habit of opening Honeycomb as the first step when investigating test failures.

Read more

Start now free