# Observability (/configure/reference/observability)



KubeMQ reports on itself through three domains: **OpenTelemetry** traces and metrics, an
**audit** log of operations, and **notifications** about server events. Each setting is
shown for both deployment targets — Docker single-node (`config.yaml` key · env var) and
Kubernetes/Helm (`spec.*` path). A dash (`—`) in the Helm/CRD column means the setting is
not available on that surface; a dash in the env-var slot means the field has **no
environment binding at all** (config-file/secret-only).

<Callout type="info">
  This page is the **settings reference*&#x2A; for telemetry, audit, and notifications. For
  concepts, how-to guides, the full metric series, the audit event catalog, and the
  management API, see &#x2A;*[Observability](/operate/observability)**.
</Callout>

<Callout type="warn">
  **Version floor:** the `spec.telemetry.*` and `spec.audit.*` fields are present throughout
  the current GA chart line — `kubemq-crds` and `kubemq-cluster` **3.x** (latest **3.2.0**)
  with `kubemq-controller` **2.x** (operator **v2.3.0**). Anything older than the 3.0.0 /
  2.0.0 GA release predates this reference and will reject these fields; upgrade to the
  current line. On Docker the `telemetry.*` and `audit.*` keys are available regardless of
  chart version.
</Callout>

## Telemetry (OpenTelemetry) [#telemetry-opentelemetry]

KubeMQ exports traces and metrics over OTLP. The scalar fields are a first-class
`spec.telemetry.*` group on Kubernetes. The master `enable` flag is **off by default**;
turn it on, then optionally tune the traces, metrics, and exporter sub-blocks. Every
sub-block only takes effect while the master switch is on.

| Setting                 | Type               | Default          | Valid values                                                   | Docker (config.yaml key · env var)                                                            | Helm/CRD path                           | Notes                                                                                                                                  |
| ----------------------- | ------------------ | ---------------- | -------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| Enable (master)         | bool               | `false`          | true / false                                                   | `telemetry.enable` · `TELEMETRY_ENABLE`                                                       | `spec.telemetry.enable`                 | Master switch — off disables all telemetry (`Validate` returns early).                                                                 |
| Service name            | string             | `kubemq`         | non-empty service name                                         | `telemetry.servicename` · `TELEMETRY_SERVICE_NAME`                                            | `spec.telemetry.serviceName`            | Reported as the OTLP `service.name`. Empty is coerced back to `kubemq` at load.                                                        |
| Traces enable           | bool               | `true`           | true / false                                                   | `telemetry.traces.enable` · `TELEMETRY_TRACES_ENABLE`                                         | `spec.telemetry.traces.enable`          | Applies only when the master switch is on.                                                                                             |
| Traces sampling ratio   | float64            | `1.0`            | `0`–`1`                                                        | `telemetry.traces.samplingratio` · `TELEMETRY_TRACES_SAMPLING_RATIO`                          | `spec.telemetry.traces.samplingRatio`   | Fraction of traces sampled. Out-of-range values are coerced back to `1.0`.                                                             |
| Traces sampler          | enum               | `parent_based`   | `always_on` / `always_off` / `trace_id_ratio` / `parent_based` | `telemetry.traces.sampler` · `TELEMETRY_TRACES_SAMPLER`                                       | `spec.telemetry.traces.sampler`         | Sampling strategy. Any other value is coerced to `parent_based`.                                                                       |
| Metrics enable          | bool               | `true`           | true / false                                                   | `telemetry.metrics.enable` · `TELEMETRY_METRICS_ENABLE`                                       | `spec.telemetry.metrics.enable`         | Applies only when the master switch is on.                                                                                             |
| Metrics export interval | duration           | `30s`            | Go duration string                                             | `telemetry.metrics.exportinterval` · `TELEMETRY_METRICS_EXPORT_INTERVAL`                      | `spec.telemetry.metrics.exportInterval` | How often metrics are pushed. Unparseable values are coerced to `30s`.                                                                 |
| Exporter protocol       | enum               | `grpc`           | `grpc` / `http`                                                | `telemetry.exporter.protocol` · `TELEMETRY_EXPORTER_PROTOCOL`                                 | `spec.telemetry.exporter.protocol`      | OTLP wire protocol. Any other value is coerced to `grpc`.                                                                              |
| Exporter endpoint       | string             | `localhost:4317` | host:port                                                      | `telemetry.exporter.endpoint` · `TELEMETRY_EXPORTER_ENDPOINT`                                 | `spec.telemetry.exporter.endpoint`      | OTLP collector endpoint. Empty is coerced to `localhost:4317`.                                                                         |
| Exporter insecure       | bool               | `true`           | true / false                                                   | `telemetry.exporter.insecure` · `TELEMETRY_EXPORTER_INSECURE`                                 | `spec.telemetry.exporter.insecure`      | Disable TLS to the collector.                                                                                                          |
| Exporter compression    | enum               | `gzip`           | `gzip` / `none`                                                | `telemetry.exporter.compression` · `TELEMETRY_EXPORTER_COMPRESSION`                           | `spec.telemetry.exporter.compression`   | Payload compression. Any other value is coerced to `gzip`.                                                                             |
| Exporter timeout        | duration           | `10s`            | Go duration string                                             | `telemetry.exporter.timeout` · `TELEMETRY_EXPORTER_TIMEOUT`                                   | `spec.telemetry.exporter.timeout`       | Export request timeout. Unparseable values are coerced to `10s`.                                                                       |
| Exporter headers        | map\[string]string | `{}`             | key/value map                                                  | `telemetry.exporter.headers` · ⚠️ `TELEMETRY_EXPORTER_HEADERS` (**do not use — see callout**) | —                                       | **Config-file-only in practice.** A `TELEMETRY_EXPORTER_HEADERS` env binding exists but is a landmine (below). Not exposed on the CRD. |
| Resource attributes     | map\[string]string | `{}`             | key/value map                                                  | `telemetry.resource` · — (no env binding)                                                     | —                                       | **Config-file-only.** No `bindViperEnv` call and no CRD field — set only in the mounted `config.yaml`.                                 |

<Callout type="warn">
  **Do not set `TELEMETRY_EXPORTER_HEADERS` — it corrupts the entire running config.**
  `Telemetry.Exporter.Headers` is a `map[string]string`, and viper/mapstructure has no
  default string→map decode hook. Setting the env var makes `viper.Unmarshal` fail, and the
  loader's error path **discards the whole config and falls back to pure defaults** — every
  other file setting and env var is silently lost, with only a stderr line as evidence. The
  env name appears in the golden key list, but treat it as unusable: configure exporter
  headers only via the `telemetry.exporter.headers` map in the mounted `config.yaml`. The
  sibling `telemetry.resource` map is safer — it has **no** env binding at all, so it is
  cleanly config-file-only. Neither map has a CRD path.
</Callout>

## Audit [#audit]

The audit log records server operations and retains them for a configurable window. It is a
first-class `spec.audit.*` group on Kubernetes and is **enabled by default**. The CRD models
`enable` as `*bool` deliberately — because audit defaults on server-side, a plain bool could
never emit `false` to turn it off.

| Setting                | Type | Default | Valid values | Docker (config.yaml key · env var)                                | Helm/CRD path                       | Notes                                                                                                                     |
| ---------------------- | ---- | ------- | ------------ | ----------------------------------------------------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| Enable                 | bool | `true`  | true / false | `audit.enable` · `AUDIT_ENABLE`                                   | `spec.audit.enable`                 | Audit is on by default. CRD field is `*bool` so it can emit `false`.                                                      |
| Retention hours        | int  | `720`   | ≥ 1          | `audit.retentionhours` · `AUDIT_RETENTION_HOURS`                  | `spec.audit.retentionHours`         | How long audit records are kept (720h = 30 days). Values \< 1 are coerced to `720`. CRD type is `int32` with `Minimum=1`. |
| Cleanup interval (min) | int  | `60`    | ≥ 1          | `audit.cleanupintervalminutes` · `AUDIT_CLEANUP_INTERVAL_MINUTES` | `spec.audit.cleanupIntervalMinutes` | How often expired records are purged. Values \< 1 are coerced to `60`. CRD type is `int32` with `Minimum=1`.              |

## Notifications [#notifications]

Server-event notifications are published to an internal channel. The toggle and prefix have
**name divergences** between the two targets — read the Notes column carefully. The server
struct has no `mapstructure` tags; viper binds these by field path (case-insensitive).

| Setting           | Type   | Default         | Valid values     | Docker (config.yaml key · env var)                                        | Helm/CRD path               | Notes                                                  |
| ----------------- | ------ | --------------- | ---------------- | ------------------------------------------------------------------------- | --------------------------- | ------------------------------------------------------ |
| Enable            | bool   | `false`         | true / false     | `notification.enable` · `NOTIFICATION_ENABLE`                             | `spec.notification.enabled` | **Name divergence:** Docker `enable` ↔ Helm `enabled`. |
| Channel prefix    | string | `notifications` | non-empty prefix | `notification.reportchannelprefix` · `NOTIFICATION_REPORT_CHANNEL_PREFIX` | `spec.notification.prefix`  | **Name divergence:** `reportchannelprefix` ↔ `prefix`. |
| Log notifications | bool   | `false`         | true / false     | `notification.log` · `NOTIFICATION_LOG`                                   | `spec.notification.log`     | Also write notifications to the server log.            |

<Callout type="warn">
  **Cross-field rule:** when notifications are enabled, if `log` is `false` **and** the
  channel prefix is empty, the server fails validation with
  `bad notification configuration: missing parameters`. Either keep a non-empty prefix
  (the default `notifications` satisfies this) or set `log: true`.
</Callout>

## Example [#example]

Enable telemetry on each target. This is a single-setting snippet — see the
[Docker guide](/configure/docker) and the
[Kubernetes guide](/configure/kubernetes) for complete, runnable configurations.

<Tabs items="[&#x22;Docker&#x22;, &#x22;Helm&#x22;]">
  <Tab value="Docker">
    ```yaml title="config.yaml"
    telemetry:
      enable: true
    ```
  </Tab>

  <Tab value="Helm">
    ```yaml title="values.yaml"
    telemetry:
      enable: true
    ```
  </Tab>
</Tabs>

For the full Docker delivery methods (env vars, mounted `config.yaml`, the `CONFIG`
variable) see the [Docker guide](/configure/docker); for `values.yaml` mapped to
the `KubemqCluster` spec see the [Kubernetes guide](/configure/kubernetes).