# Prometheus Metrics (/operate/observability/metrics)



KubeMQ exposes an always-on Prometheus exporter that reports message counts, byte volumes,
client counts, queue health, RPC latency, cluster state, and agent-platform activity. It
needs no configuration — it is running the moment KubeMQ starts.

## Overview [#overview]

The Prometheus exporter is **always on**. The same in-process metrics also power the
built-in dashboard and its snapshot system, so the numbers
you scrape with Prometheus and the numbers you see in the dashboard come from one source.

Metrics are served in-process by the [management API](/operate/observability/api-reference) at
`GET /metrics` on the management API port (`:8080`) — the same port that serves health
probes, stats, and the dashboard. There is no separate listener and no extra setup; the
endpoint returns standard Prometheus text format.

<Callout type="info">
  Prometheus is the **pull** path and is always on. KubeMQ also supports an opt-in
  **push** path — OpenTelemetry metrics over OTLP — covered on the
  [Distributed Tracing](/operate/observability/tracing) page. The OTel `pending` instrument
  (active in-flight messages) is exported only over OTLP; there is no equivalent Prometheus
  series.
</Callout>

## Scraping KubeMQ [#scraping-kubemq]

Read the endpoint directly with `curl`:

```bash
curl http://localhost:8080/metrics
```

Point Prometheus at the same endpoint with a `scrape_config`:

```yaml title="prometheus.yml"
scrape_configs:
  - job_name: kubemq
    metrics_path: /metrics
    static_configs:
      - targets: ["kubemq:8080"]
```

## Labels [#labels]

Core messaging metrics carry a shared label set. The `type` and `side` label values are
determined by the messaging pattern.

| Label       | Values                                                    | Description                                                                                                                                                                                                            |
| ----------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `node`      | host name                                                 | The KubeMQ node that recorded the metric.                                                                                                                                                                              |
| `client_id` | client id                                                 | The connected client. **Aggregated away on store series** — the per-channel store collectors drop `client_id` at collect time, so queue, RPC-latency, and similar series carry only `node`, `type`, `side`, `channel`. |
| `type`      | `events`, `events_store`, `commands`, `queries`, `queues` | The messaging pattern.                                                                                                                                                                                                 |
| `side`      | `send`, `receive`                                         | Whether the metric is for the producing or consuming side of the channel.                                                                                                                                              |
| `channel`   | channel name                                              | The message channel the metric belongs to.                                                                                                                                                                             |

## Core messaging metrics [#core-messaging-metrics]

These collectors track message activity across all five patterns. They share the label set
above.

| Prometheus name             | Type    | Description                               |
| --------------------------- | ------- | ----------------------------------------- |
| `kubemq_messages_count`     | Counter | Total messages sent/received per channel. |
| `kubemq_messages_volume`    | Counter | Total byte volume per channel.            |
| `kubemq_messages_last_seen` | Gauge   | Unix timestamp (ms) of last activity.     |
| `kubemq_messages_delayed`   | Counter | Number of delayed queue messages.         |
| `kubemq_messages_expired`   | Counter | Total expired queue messages.             |
| `kubemq_messages_waiting`   | Gauge   | Queue messages waiting for consumers.     |
| `kubemq_clients_count`      | Gauge   | Connected client count per channel.       |
| `kubemq_errors_count`       | Counter | Total errors per channel.                 |
| `kubemq_messages_responses` | Counter | Total RPC responses per channel.          |

## Queue health metrics [#queue-health-metrics]

These series give queue operators visibility into redelivery, dead-letter dispositions,
backlog age, and dwell time. They use the standard four-label set (`node`, `type`, `side`,
`channel`) — `client_id` is aggregated away — except where a row notes an extra label.

| Prometheus name                     | Type    | Description                                                                                                                                                                                                                                                                                                                                                            |
| ----------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `kubemq_queue_redelivery_total`     | Counter | Total queue redeliveries per channel — a receive whose post-increment receive count is greater than one — attributed to the source queue channel on the `receive` side. Uses the standard four labels only (no extra label).                                                                                                                                           |
| `kubemq_queue_dlq_total`            | Counter | Total queue dead-letter dispositions per channel, attributed to the source channel on the `receive` side. Carries one **extra `reason` label**: `routed` (re-published to the configured dead-letter queue) or `dropped` (no DLQ configured or a marshal error — silent data loss made visible). The `reason` label is on `dlq_total` only, not on `redelivery_total`. |
| `kubemq_queue_backlog_age_seconds`  | Gauge   | Age (seconds) of the oldest undelivered message in the backlog, per channel (`queues`/`receive`). A live gauge recomputed each 5-second snapshot cycle; `0` when the backlog is empty or fully delivered, and `0` after a restart until the next cycle rebuilds it.                                                                                                    |
| `kubemq_queue_dwell_seconds_sum`    | Counter | Cumulative sum of per-message dwell times (seconds) on the `receive` side — the time from when a message entered the queue until it was delivered to a consumer. Survives restarts. Divide by `kubemq_queue_dwell_messages_total` for the average dwell.                                                                                                               |
| `kubemq_queue_dwell_messages_total` | Counter | Cumulative count of messages whose dwell time was recorded. Survives restarts. The denominator for the average dwell time.                                                                                                                                                                                                                                             |

## RPC latency metrics [#rpc-latency-metrics]

These series measure command and query round-trip latency on the `send` side. They use the
standard four-label set.

| Prometheus name                  | Type    | Description                                                                                                                                                                                                                                                                                                                                                |
| -------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `kubemq_rpc_latency_seconds_sum` | Counter | Cumulative sum of RPC round-trip times (seconds) on the `send` side of a command or query channel — measured from request dispatch until the response is received. Survives restarts. Recorded only when a response is received (errored or timed-out requests with no response are excluded). Divide by `kubemq_rpc_latency_calls_total` for the average. |
| `kubemq_rpc_latency_calls_total` | Counter | Cumulative count of RPC calls whose round-trip latency was recorded (`type ∈ {commands, queries}`, `side = send`). Survives restarts. The denominator for the average RPC latency.                                                                                                                                                                         |

<Callout type="info">
  **Cache-served queries are excluded.** The query cache short-circuits before the latency is
  recorded, so these series measure real backend round-trips (cache misses), not local cache
  lookups. Commands have no cache and are always recorded.
</Callout>

## Latency histograms [#latency-histograms]

Three histogram instruments expose latency **distributions** (so you can compute percentiles
with `histogram_quantile`), labeled by `node`, `type`, and `side`.

| Prometheus name                        | Type      | Description                                                                                                                                                                                                           |
| -------------------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `kubemq_rpc_latency_histogram_seconds` | Histogram | RPC round-trip latency distribution (commands and queries, `send` side). Same gate as `kubemq_rpc_latency_seconds_sum` — recorded only when a response is received; cache-served queries are excluded.                |
| `kubemq_queue_dwell_histogram_seconds` | Histogram | Queue dwell-time distribution (`queues`, `receive` side). Its population matches `kubemq_queue_dwell_messages_total`.                                                                                                 |
| `kubemq_operation_duration_seconds`    | Histogram | Per-operation send duration for patterns with no round-trip: `type ∈ {events, events_store, queues}` on the `send` side. Commands and queries use `kubemq_rpc_latency_histogram_seconds` for their percentile source. |

All three histograms share one bucket set (seconds), giving a consistent latency vocabulary
across `/metrics`:

```text
0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60
```

## Cluster & infra metrics [#cluster--infra-metrics]

These gauges expose node health, cluster role, and configured peer state, labeled by `node`
(and `role` where noted).

| Prometheus name                | Type  | Description                                                                                                                                                                                                                                 |
| ------------------------------ | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `kubemq_cluster_leader`        | Gauge | `1` if this node is the cluster leader **or** a standalone node (both act as the authoritative message processor); `0` otherwise.                                                                                                           |
| `kubemq_cluster_role`          | Gauge | Info-gauge: emits `1` for the active `role` label only, `0` for the rest. `role ∈ {leader, follower, candidate, standalone}`. Filter with `kubemq_cluster_role == 1`.                                                                       |
| `kubemq_cluster_ready`         | Gauge | `1` when the node is ready to accept traffic; `0` during startup or unhealthy states.                                                                                                                                                       |
| `kubemq_cluster_healthy`       | Gauge | `1` when the node is healthy. Distinct from ready — a node can be healthy but not yet ready.                                                                                                                                                |
| `kubemq_cluster_state_seconds` | Gauge | Seconds since the last cluster state change. For clustered nodes: seconds since the last ready-settle (leadership election or follower sync). For standalone nodes: seconds since server startup.                                           |
| `kubemq_cluster_peers`         | Gauge | The configured bootstrap peer count, including self (minimum `1` for standalone). This is the configured bootstrap peer list, **not** live cluster membership — on a running cluster, live membership may differ during leader transitions. |

On a single-node (standalone) server, `kubemq_cluster_leader = 1`,
`kubemq_cluster_role{role="standalone"} = 1`, `kubemq_cluster_ready = 1` once ready, and
`kubemq_cluster_peers = 1`. On a multi-node cluster, exactly one node has
`kubemq_cluster_leader = 1` at any given time.

## Agent-platform metrics [#agent-platform-metrics]

KubeMQ's [agent platform](/aiway) emits its own series for MCP tool calls and A2A agent
traffic. These use domain-specific labels rather than the core messaging label set.

| Prometheus name                        | Type      | Labels                         | Description                                                                                                     |
| -------------------------------------- | --------- | ------------------------------ | --------------------------------------------------------------------------------------------------------------- |
| `kubemq_mcp_tool_calls_total`          | Counter   | `tool`, `status`               | Total MCP tool calls.                                                                                           |
| `kubemq_mcp_tool_duration_seconds`     | Histogram | `tool`                         | Duration of MCP tool calls (seconds).                                                                           |
| `kubemq_a2a_requests_total`            | Counter   | `agent_id`, `method`, `status` | Total A2A requests routed to agents (any path — HTTP gateway and MCP bridge).                                   |
| `kubemq_a2a_request_duration_seconds`  | Histogram | `agent_id`                     | Duration of A2A requests routed to agents (any path — HTTP gateway and MCP bridge).                             |
| `kubemq_a2a_errors_total`              | Counter   | `agent_id`, `error_code`       | Total A2A errors for requests routed to agents (any path — HTTP gateway and MCP bridge).                        |
| `kubemq_a2a_stream_events_total`       | Counter   | `agent_id`                     | Total A2A stream events delivered to SSE clients (receive side).                                                |
| `kubemq_a2a_stream_outcomes_total`     | Counter   | `agent_id`, `outcome`          | Total A2A streams by terminal `outcome` (`done`, `error`, `idle_timeout`, `canceled`) — exactly one per stream. |
| `kubemq_a2a_registry_operations_total` | Counter   | `op`, `status`                 | Total agent registry operations (register, deregister, heartbeat, expire).                                      |
| `kubemq_a2a_sse_streams_active`        | Gauge     | —                              | Current number of active A2A SSE streams.                                                                       |

The MCP tool-call histogram (`kubemq_mcp_tool_duration_seconds`) uses the bucket set
`0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10`; the A2A request-duration histogram
(`kubemq_a2a_request_duration_seconds`) uses `0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60`.

<Callout type="info">
  **The `agent_id` label is bounded.** The first \~1000 distinct agent IDs each get their own
  label value; every subsequent agent ID collapses into `agent_id="other"`. This is cardinality
  protection — for exact per-agent leaderboards across the whole fleet, use the dashboard rather
  than Prometheus.
</Callout>

<Callout type="info">
  **The `method` label is sanitized.** Each A2A method is checked against a fixed allowlist of
  known methods; anything not in the list is recorded as `method="unknown"`. This keeps a
  malformed or malicious request stream from exploding Prometheus label cardinality.
</Callout>

In a cluster, these counters are aggregated across all nodes, so the totals reflect the whole
deployment rather than a single replica.

## PromQL examples [#promql-examples]

```text
# RPC p99 latency (commands) — rolling 5-minute window
histogram_quantile(0.99,
  rate(kubemq_rpc_latency_histogram_seconds_bucket{type="commands"}[5m])
)

# Queue dwell p95 across all nodes
histogram_quantile(0.95,
  sum(rate(kubemq_queue_dwell_histogram_seconds_bucket[5m])) by (le)
)

# Alert: no leader exists across the cluster
sum(kubemq_cluster_leader) == 0

# Alert: a node has been not-ready for more than 60 seconds
kubemq_cluster_ready == 0 and kubemq_cluster_state_seconds > 60
```

## Grafana [#grafana]

Point Grafana at the Prometheus instance scraping KubeMQ and build dashboards on the series
above — message rates from `kubemq_messages_count`, queue health from the
`kubemq_queue_*` series, latency percentiles from the histogram buckets, and cluster state
from the `kubemq_cluster_*` gauges.

## Internal channels [#internal-channels]

Internal cluster channels are filtered out of the user-facing metrics, so they never appear
in the series above.