# Distributed Tracing (/operate/observability/tracing)



KubeMQ integrates with [OpenTelemetry](https://opentelemetry.io/) (OTel) for distributed
tracing and metrics export. It is **opt-in** — off by default and turned on with a single
`telemetry.enable` flag. When enabled, KubeMQ instruments every messaging operation (Events,
Events Store, Queues, Commands, Queries) with spans and exports them, along with a set of OTel
metric instruments, over the OTLP protocol to any compatible backend (Jaeger, Grafana Tempo,
Datadog, and others).

Tracing is **complementary** to the [always-on Prometheus exporter](/operate/observability/metrics).
Prometheus gives you a pull-based metric series for dashboards and alerting; OpenTelemetry adds
distributed traces with context propagation across services, plus a parallel push-based metric
stream over OTLP.

## Enable it [#enable-it]

Telemetry is controlled by the master `telemetry.enable` switch (off by default). Turn it on,
then tune the traces, metrics, and exporter sub-blocks as needed.

<Tabs items="[&#x22;Docker&#x22;, &#x22;Helm&#x22;]">
  <Tab value="Docker">
    ```yaml title="config.yaml"
    telemetry:
      enable: true
    ```
  </Tab>

  <Tab value="Helm">
    ```yaml title="values.yaml"
    telemetry:
      enable: true
    ```
  </Tab>
</Tabs>

<Callout type="warn">
  **Version floor:** the `spec.telemetry.*` fields are present throughout the current GA
  chart line — `kubemq-crds` and `kubemq-cluster` **3.x** (latest **3.2.0**) with
  `kubemq-controller` **2.x** (operator **v2.3.0**). Anything older than the 3.0.0 / 2.0.0 GA
  release predates this reference and will reject these fields; upgrade to the current line.
  On Docker the `telemetry.*` keys are available regardless of chart version.
</Callout>

The full set of telemetry settings — service name, traces/metrics sub-blocks, exporter
options, and the Docker/Helm key and environment-variable mappings — lives in the
[Configuration reference](/configure/reference/observability). This page covers what
those settings do and what you get when telemetry is on.

## OTLP exporter [#otlp-exporter]

Traces and metrics are exported over OTLP to a collector or backend. The exporter is configured
under `telemetry.exporter`.

| Setting       | Default          | Description                                                                                                                                                  |
| ------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `protocol`    | `grpc`           | OTLP wire protocol — `grpc` (default endpoint `:4317`) or `http` (default endpoint `:4318`).                                                                 |
| `endpoint`    | `localhost:4317` | OTLP collector endpoint (`host:port`).                                                                                                                       |
| `insecure`    | `true`           | When `true`, the connection skips TLS. Set `false` to use TLS 1.2+ to the collector.                                                                         |
| `compression` | `gzip`           | Payload compression — `gzip` or `none`.                                                                                                                      |
| `timeout`     | `10s`            | Export request timeout (Go duration).                                                                                                                        |
| `headers`     | `{}`             | Custom headers sent with each export request (a `key/value&#x60; map). **`config.yaml`-only** — there is no environment-variable or CRD path for this field. |

| Protocol | Default endpoint | Notes                                              |
| -------- | ---------------- | -------------------------------------------------- |
| `grpc`   | `localhost:4317` | OTLP over gRPC. Supports TLS and gzip compression. |
| `http`   | `localhost:4318` | OTLP over HTTP. Supports TLS and gzip compression. |

## Samplers [#samplers]

The sampler decides which traces are recorded. It is set with `telemetry.traces.sampler`; the
ratio-based samplers read `telemetry.traces.samplingRatio` (a fraction from `0.0` to `1.0`).

| Sampler          | Description                                                                                          |
| ---------------- | ---------------------------------------------------------------------------------------------------- |
| `always_on`      | Sample every trace.                                                                                  |
| `always_off`     | Sample no traces.                                                                                    |
| `trace_id_ratio` | Sample a fraction of traces based on `samplingRatio` (e.g. `0.1` = 10% of traces).                   |
| `parent_based`   | Use the parent span's sampling decision; falls back to `trace_id_ratio` for root spans. The default. |

## Environment variable overrides [#environment-variable-overrides]

Standard OTel environment variables are read after the config file loads and **override** the
file values. These are the recommended way to set exporter and sampler options from outside the
container.

| Environment variable             | Overrides             | Notes                                                                                                                                         |
| -------------------------------- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `OTEL_SERVICE_NAME`              | Service name          | The OTLP `service.name` resource attribute.                                                                                                   |
| `OTEL_EXPORTER_OTLP_ENDPOINT`    | Exporter endpoint     |                                                                                                                                               |
| `OTEL_EXPORTER_OTLP_PROTOCOL`    | Exporter protocol     | `grpc` or `http`.                                                                                                                             |
| `OTEL_EXPORTER_OTLP_INSECURE`    | Exporter insecure     | `"true"` / `"false"`.                                                                                                                         |
| `OTEL_EXPORTER_OTLP_COMPRESSION` | Exporter compression  | `gzip` or `none`.                                                                                                                             |
| `OTEL_EXPORTER_OTLP_TIMEOUT`     | Exporter timeout      | Integer milliseconds (e.g. `"10000"`).                                                                                                        |
| `OTEL_EXPORTER_OTLP_HEADERS`     | Exporter headers      | Comma-separated `key=value` pairs.                                                                                                            |
| `OTEL_TRACES_SAMPLER`            | Traces sampler        | OTel sampler names: `always_on`, `always_off`, `traceidratio`, `parentbased_traceidratio`, `parentbased_always_on`, `parentbased_always_off`. |
| `OTEL_TRACES_SAMPLER_ARG`        | Traces sampling ratio | Float string (e.g. `"0.1"`).                                                                                                                  |

<Callout type="info">
  KubeMQ also derives its own `TELEMETRY_*` environment variables from the config keys (for
  example `TELEMETRY_ENABLE`, `TELEMETRY_EXPORTER_ENDPOINT`). Where both exist, **prefer the
  standard `OTEL_*` variables** above — they are the conventional way to configure an OTel
  exporter and are read after the KubeMQ-specific values.
</Callout>

## What gets traced [#what-gets-traced]

When tracing is on, every messaging pattern is instrumented. Each operation produces a span
whose name encodes the operation and channel.

| Span name                        | Span kind | Pattern          |
| -------------------------------- | --------- | ---------------- |
| `process events.{channel}`       | Consumer  | Events           |
| `process events-store.{channel}` | Consumer  | Events Store     |
| `process commands.{channel}`     | Consumer  | Commands         |
| `process queries.{channel}`      | Consumer  | Queries          |
| `process queues.{channel}`       | Consumer  | Queues (single)  |
| `publish-batch queues.{channel}` | Consumer  | Queues (batch)   |
| `deliver queues.{channel}`       | Producer  | Queues (receive) |

In addition, the shared HTTP server applies automatic **transport-level tracing** to all
HTTP-based connectors (REST, A2A, MCP, and CloudEvents) under the service name `kubemq-http`.
When telemetry is enabled, each HTTP request handled by these connectors generates a span with
the standard HTTP attributes:

| Attribute          | Description                    |
| ------------------ | ------------------------------ |
| `http.method`      | HTTP method (GET, POST, etc.). |
| `http.route`       | Matched route pattern.         |
| `http.status_code` | Response status code.          |
| `http.target`      | Request path.                  |

<Callout type="info">
  The A2A and MCP connectors and the agent registry have **no application-level OTel
  instrumentation** — they rely on this transport-level tracing for HTTP spans (method, route,
  status, duration) and on [Prometheus metrics](/operate/observability/metrics) for per-operation
  detail. Per-operation spans (which agent was invoked, tool execution time, JSON-RPC method)
  are not currently traced.
</Callout>

## Span attributes [#span-attributes]

Every messaging span carries a standard set of attributes:

| Attribute                       | Description                    | Example                                                   |
| ------------------------------- | ------------------------------ | --------------------------------------------------------- |
| `messaging.system`              | Always `kubemq`.               | `kubemq`                                                  |
| `messaging.operation.type`      | Operation type.                | `process`, `receive`                                      |
| `messaging.destination.name`    | Channel name.                  | `orders`                                                  |
| `messaging.message.id`          | Message, event, or request ID. | `abc123`                                                  |
| `messaging.client.id`           | Client identifier.             | `my-producer`                                             |
| `messaging.kubemq.channel_type` | Channel type.                  | `events`, `events_store`, `commands`, `queries`, `queues` |

Queue spans add these extra attributes:

| Attribute                            | Description                                                            |
| ------------------------------------ | ---------------------------------------------------------------------- |
| `messaging.kubemq.expiration`        | Message expiration (e.g. `30s`).                                       |
| `messaging.kubemq.delay`             | Message delay (e.g. `10s`).                                            |
| `messaging.kubemq.max_receive`       | Max receive count before a message is routed to the dead-letter queue. |
| `messaging.batch.message_count`      | Number of messages in a batch.                                         |
| `messaging.kubemq.batch_error_count` | Number of failed messages in a batch.                                  |
| `messaging.kubemq.messages_received` | Messages received in a receive operation.                              |
| `messaging.kubemq.messages_expired`  | Messages expired during a receive.                                     |

## Context propagation [#context-propagation]

KubeMQ propagates trace context through message tags using the W3C **TraceContext** and
**Baggage** propagators:

1. **Inject** — when a message passes through the tracing middleware, the current trace context
   is injected into the message's `Tags` map.
2. **Extract** — when the message arrives at the message broker, the trace headers are read back
   out of the `Tags` map to link the consumer span to the producer.

This enables end-to-end tracing across **producer → the message broker → consumer**, even when
the producer and consumer use different SDKs.

## Error recording [#error-recording]

When a send or receive operation fails, the tracing middleware:

1. Records the error on the span.
2. Sets the span status to `Error`.
3. Increments the `messaging.kubemq.errors.count` metric (see below).

## OTel metrics [#otel-metrics]

In addition to traces, KubeMQ exports a set of OTel metric instruments over OTLP. These are a
parallel push-based path to the [pull-based Prometheus series](/operate/observability/metrics) and
are useful when your backend already ingests OTLP metrics (Mimir, Datadog, and others). All
instruments share the labels `node`, `type` (channel type), `side` (`send` / `receive`), and
`channel`.

| Instrument                            | Type          | Unit  | Description                    |
| ------------------------------------- | ------------- | ----- | ------------------------------ |
| `messaging.kubemq.messages.count`     | Counter       | —     | Total messages processed.      |
| `messaging.kubemq.messages.volume`    | Counter       | bytes | Total message volume in bytes. |
| `messaging.kubemq.errors.count`       | Counter       | —     | Total errors.                  |
| `messaging.kubemq.messages.pending`   | UpDownCounter | —     | Pending messages.              |
| `messaging.kubemq.messages.delayed`   | UpDownCounter | —     | Delayed queue messages.        |
| `messaging.kubemq.messages.expired`   | Counter       | —     | Expired messages.              |
| `messaging.kubemq.messages.waiting`   | UpDownCounter | —     | Queue messages waiting.        |
| `messaging.kubemq.clients.count`      | UpDownCounter | —     | Connected clients.             |
| `messaging.kubemq.messages.last_seen` | Gauge         | —     | Last activity timestamp.       |
| `messaging.kubemq.responses.count`    | Counter       | —     | RPC responses.                 |
| `messaging.kubemq.operation.duration` | Histogram     | s     | Per-operation latency.         |
| `messaging.kubemq.message.size`       | Histogram     | bytes | Message payload size.          |
| `messaging.kubemq.queue.depth`        | UpDownCounter | —     | Queue depth per channel.       |

<Callout type="warn">
  The OTel `messaging.kubemq.operation.duration` histogram uses its **own** bucket boundaries
  (in seconds), which are **distinct** from the Prometheus histogram buckets on the
  [Metrics page](/operate/observability/metrics) — do not assume they match:

  ```text
  0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10
  ```
</Callout>

## Export to a backend [#export-to-a-backend]

Point the exporter at any OTLP-compatible collector or backend. The example below runs Jaeger
locally and exports traces to it over OTLP gRPC.

<Tabs items="[&#x22;Jaeger (Docker Compose)&#x22;, &#x22;KubeMQ config&#x22;]">
  <Tab value="Jaeger (Docker Compose)">
    ```yaml title="docker-compose.yml"
    services:
      jaeger:
        image: jaegertracing/all-in-one:latest
        ports:
          - "4317:4317"     # OTLP gRPC
          - "16686:16686"   # Jaeger UI
        environment:
          - COLLECTOR_OTLP_ENABLED=true
    ```
  </Tab>

  <Tab value="KubeMQ config">
    ```yaml title="config.yaml"
    telemetry:
      enable: true
      serviceName: "kubemq-production"
      traces:
        enable: true
        sampler: "parent_based"
        samplingRatio: 0.1      # sample 10% of traces
      metrics:
        enable: true
        exportInterval: "30s"
      exporter:
        protocol: "grpc"
        endpoint: "jaeger:4317"
        insecure: true
        compression: "gzip"
    ```
  </Tab>
</Tabs>

The same exporter block works for **Grafana Tempo** and **Datadog** — point `endpoint` at the
relevant OTLP receiver, switch `protocol` to `http` if the backend expects OTLP/HTTP, and supply
any required auth headers via `OTEL_EXPORTER_OTLP_HEADERS`.

## Correlate with logs [#correlate-with-logs]

When tracing is on, KubeMQ adds `trace_id` and `span_id` fields to the log lines emitted during
a traced operation, so you can pivot between a log line and its trace in your backend. See
[Structured Logging](/operate/observability/logging) for the log field set and a Grafana
Loki ↔ Tempo derived-fields setup.

## Related [#related]

<Cards>
  <Card title="Prometheus Metrics" href="/operate/observability/metrics" description="The always-on pull-based metric series, complementary to OTel metrics." />

  <Card title="Structured Logging" href="/operate/observability/logging" description="JSON logs on stdout, correlated with traces via trace and span IDs." />

  <Card title="Telemetry settings reference" href="/configure/reference/observability" description="Every telemetry config key with Docker and Helm mappings." />
</Cards>