Distributed Tracing

KubeMQ integrates with OpenTelemetry (OTel) for distributed tracing and metrics export. It is opt-in — off by default and turned on with a single telemetry.enable flag. When enabled, KubeMQ instruments every messaging operation (Events, Events Store, Queues, Commands, Queries) with spans and exports them, along with a set of OTel metric instruments, over the OTLP protocol to any compatible backend (Jaeger, Grafana Tempo, Datadog, and others).

Tracing is complementary to the always-on Prometheus exporter. Prometheus gives you a pull-based metric series for dashboards and alerting; OpenTelemetry adds distributed traces with context propagation across services, plus a parallel push-based metric stream over OTLP.

Enable it

Telemetry is controlled by the master telemetry.enable switch (off by default). Turn it on, then tune the traces, metrics, and exporter sub-blocks as needed.

telemetry:
  enable: true

telemetry:
  enable: true

The full set of telemetry settings — service name, traces/metrics sub-blocks, exporter options, and the Docker/Helm key and environment-variable mappings — lives in the Configuration reference. This page covers what those settings do and what you get when telemetry is on.

OTLP exporter

Traces and metrics are exported over OTLP to a collector or backend. The exporter is configured under telemetry.exporter.

Samplers

The sampler decides which traces are recorded. It is set with telemetry.traces.sampler; the ratio-based samplers read telemetry.traces.samplingRatio (a fraction from 0.0 to 1.0).

Environment variable overrides

Standard OTel environment variables are read after the config file loads and override the file values. These are the recommended way to set exporter and sampler options from outside the container.

What gets traced

When tracing is on, every messaging pattern is instrumented. Each operation produces a span whose name encodes the operation and channel.

In addition, the shared HTTP server applies automatic transport-level tracing to all HTTP-based connectors (REST, A2A, MCP, and CloudEvents) under the service name kubemq-http. When telemetry is enabled, each HTTP request handled by these connectors generates a span with the standard HTTP attributes:

Span attributes

Every messaging span carries a standard set of attributes:

Queue spans add these extra attributes:

Context propagation

KubeMQ propagates trace context through message tags using the W3C TraceContext and Baggage propagators:

1. Inject — when a message passes through the tracing middleware, the current trace context is injected into the message's Tags map.
2. Extract — when the message arrives at the message broker, the trace headers are read back out of the Tags map to link the consumer span to the producer.

This enables end-to-end tracing across producer → the message broker → consumer, even when the producer and consumer use different SDKs.

Error recording

When a send or receive operation fails, the tracing middleware:

1. Records the error on the span.
2. Sets the span status to Error.
3. Increments the messaging.kubemq.errors.count metric (see below).

OTel metrics

In addition to traces, KubeMQ exports a set of OTel metric instruments over OTLP. These are a parallel push-based path to the pull-based Prometheus series and are useful when your backend already ingests OTLP metrics (Mimir, Datadog, and others). All instruments share the labels node, type (channel type), side (send / receive), and channel.

0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10

Export to a backend

Point the exporter at any OTLP-compatible collector or backend. The example below runs Jaeger locally and exports traces to it over OTLP gRPC.

services:
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "4317:4317"     # OTLP gRPC
      - "16686:16686"   # Jaeger UI
    environment:
      - COLLECTOR_OTLP_ENABLED=true

telemetry:
  enable: true
  serviceName: "kubemq-production"
  traces:
    enable: true
    sampler: "parent_based"
    samplingRatio: 0.1      # sample 10% of traces
  metrics:
    enable: true
    exportInterval: "30s"
  exporter:
    protocol: "grpc"
    endpoint: "jaeger:4317"
    insecure: true
    compression: "gzip"

The same exporter block works for Grafana Tempo and Datadog — point endpoint at the relevant OTLP receiver, switch protocol to http if the backend expects OTLP/HTTP, and supply any required auth headers via OTEL_EXPORTER_OTLP_HEADERS.

Correlate with logs

When tracing is on, KubeMQ adds trace_id and span_id fields to the log lines emitted during a traced operation, so you can pivot between a log line and its trace in your backend. See Structured Logging for the log field set and a Grafana Loki ↔ Tempo derived-fields setup.

Related