# Observability (/operate/observability)



KubeMQ reports on itself across several complementary surfaces. An always-on Prometheus
exporter, opt-in OpenTelemetry traces and metrics, structured JSON logs, a CloudEvents
audit trail, a built-in web dashboard, and an HTTP/WebSocket management API all expose
what the server is doing right now and what it has done. This section is the operator's
home for monitoring KubeMQ in production — how to scrape metrics, wire up tracing, read
the logs, query the audit log, and consume the management API.

Each surface answers a different question: Prometheus metrics give you the quantitative
time series, tracing shows you the path of a single message across patterns, logs and the
audit trail record discrete events, the dashboard renders the live picture, and the
management API is the programmatic interface behind it all.

## The observability surfaces [#the-observability-surfaces]

| Surface               | What it gives you                                                                                                         | Where it's served                    | Always-on?  |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ | ----------- |
| Prometheus metrics    | Counters, gauges, and histograms for every messaging pattern, queues, RPC latency, cluster health, and the agent platform | `GET /metrics` (`:8080`)             | Yes         |
| OpenTelemetry tracing | Distributed traces and OTel metrics over OTLP to Jaeger, Grafana Tempo, or Datadog                                        | OTLP export to an external collector | No (opt-in) |
| Structured logging    | JSON log lines on stdout — levels, the field set, and trace correlation                                                   | stdout                               | Yes         |
| Audit logging         | A CloudEvents v1.0 trail of auth, lifecycle, and data-plane error events                                                  | `GET /api/v1/audit` (`:8080`)        | Yes         |
| Built-in dashboard    | A live web view of channels, clients, queues, the patterns, cluster topology, and agents                                  | `:8080`                              | Yes         |
| Management API        | The HTTP and WebSocket interface for health, stats, snapshots, audit query, and dashboard actions                         | `:8080`                              | Yes         |

## Push vs pull [#push-vs-pull]

KubeMQ uses a **pull-first** model for metrics and a **push-first** model for traces, and
the two can run at the same time. Logs are written to stdout and captured by your runtime.

| Signal                | Model                                                  | Consumer                            |
| --------------------- | ------------------------------------------------------ | ----------------------------------- |
| Prometheus metrics    | Pull — Prometheus scrapes `GET /metrics` on demand     | Prometheus / Grafana                |
| Dashboard snapshots   | Pull — the dashboard polls the snapshot endpoints      | Built-in dashboard                  |
| OpenTelemetry traces  | Push — the OTLP exporter pushes to a collector         | Jaeger / Tempo / Datadog            |
| OpenTelemetry metrics | Push — the meter provider exports periodically         | OTLP-compatible metrics backend     |
| Audit events          | Pull — the REST query API                              | Operator / SIEM                     |
| Logs                  | Push — JSON written to stdout, captured by the runtime | Log aggregator (Loki, Splunk, etc.) |

The Prometheus exporter always runs — it also powers the built-in dashboard. OpenTelemetry
is opt-in and is enabled through the telemetry settings.

## Where each surface is served [#where-each-surface-is-served]

The management API port (`:8080`) is the same in-process port that serves the dashboard,
the metrics endpoint, the stats and snapshot endpoints, and the audit query API. When
control-plane auth is enabled, every endpoint below requires at least the **ReadOnly**
role.

| Surface                  | Endpoint                                                       | Access (control-plane auth on) |
| ------------------------ | -------------------------------------------------------------- | ------------------------------ |
| Prometheus scrape        | `GET /metrics` (`:8080`)                                       | ReadOnly+                      |
| Dashboard snapshots      | `GET /api/snapshot`, `GET /api/cluster-snapshot`               | ReadOnly+                      |
| Audit query              | `GET /api/v1/audit`, `GET /api/v1/audit/stats`                 | ReadOnly+                      |
| Channel and client stats | `GET /v1/stats/channels`, `GET /v1/stats/clients`              | ReadOnly+                      |
| Live message monitor     | WebSocket `/api/monitor`                                       | ReadOnly+                      |
| Real-time cluster stream | WebSocket `/api/connection`                                    | ReadOnly+                      |
| Billing summary          | `GET /billing`                                                 | ReadOnly+                      |
| OTLP export              | OTLP gRPC (`:4317`) or HTTP (`:4318`) to an external collector | Collector-side                 |
| Application logs         | stdout (JSON)                                                  | —                              |

## Start here [#start-here]

<Cards>
  <Card title="Prometheus Metrics" href="/operate/observability/metrics" description="Scrape the always-on exporter — every metric series for messaging, queues, RPC latency, cluster health, and the agent platform." />

  <Card title="Distributed Tracing" href="/operate/observability/tracing" description="Export OpenTelemetry traces and metrics over OTLP to Jaeger, Grafana Tempo, or Datadog." />

  <Card title="Structured Logging" href="/operate/observability/logging" description="JSON logs on stdout — levels, runtime level changes, the field set, and trace correlation." />

  <Card title="Audit Logging" href="/operate/observability/audit" description="The CloudEvents audit trail — the event catalog, retention settings, and the REST query API." />

  <Card title="Management API" href="/operate/observability/api-reference" description="The HTTP and WebSocket management API on port 8080 — envelope, status codes, endpoints, and data models." />
</Cards>

## Configure it [#configure-it]

<Callout title="Where the settings live">
  The raw configuration keys for telemetry, audit, and server-event notifications — with
  their Docker and Helm naming, defaults, and valid values — live in the
  [Observability settings reference](/configure/reference/observability). That page
  is the source of truth for the settings; this section covers the concepts, how-to, and the
  full management API.
</Callout>