# Structured Logging (/operate/observability/logging)



KubeMQ writes **structured JSON logs to stdout**. Every log line is a single JSON object, which
makes the output easy to capture and parse with any container log aggregator — Loki, Splunk,
Datadog, Fluentd, and others — with no extra log files or rotation to manage.

## Log format [#log-format]

Every log line is one JSON object on stdout:

```json
{
  "level": "INFO",
  "time": "2026-06-29T12:34:56.789Z",
  "component": "grpc",
  "caller": "grpc/grpc.go:144",
  "msg": "starting gRPC server",
  "host": "kubemq-node-1"
}
```

The field set:

| JSON key    | Description                                                                                                     |
| ----------- | --------------------------------------------------------------------------------------------------------------- |
| `msg`       | The log message string.                                                                                         |
| `level`     | The level in ALL-CAPS: `DEBUG`, `INFO`, `WARN`, `ERROR`, `FATAL`.                                               |
| `time`      | ISO 8601 timestamp with milliseconds.                                                                           |
| `component` | The logger name (the KubeMQ component that emitted the line, e.g. `grpc`, `array`).                             |
| `caller`    | The trimmed `file:line` source location.                                                                        |
| `stack`     | A stack trace — present on `ERROR` and above when configured.                                                   |
| `host`      | The node hostname.                                                                                              |
| `trace_id`  | Trace identifier — present only when [tracing](/operate/observability/tracing) is enabled and a span is active. |
| `span_id`   | Span identifier — present only when tracing is enabled and a span is active.                                    |

Components enrich their loggers with extra context fields (such as `client_id` or `channel`)
that appear as additional key/value pairs on the relevant lines.

## Log levels [#log-levels]

KubeMQ maps six application levels onto the JSON `level` field. `Info` is the default.

| Level | JSON `level`     |
| ----- | ---------------- |
| Trace | `DEBUG`          |
| Debug | `DEBUG`          |
| Info  | `INFO` (default) |
| Warn  | `WARN`           |
| Error | `ERROR`          |
| Fatal | `FATAL`          |

Trace and Debug both emit at the `DEBUG` JSON level. The numeric values used in configuration
are `0=Trace`, `1=Debug`, `2=Info`, `3=Warn`, `4=Error`, `5=Fatal`.

## Configure the level [#configure-the-level]

Set the log level at startup with the `log.level` config key (or its environment-variable
binding). The default is `2` (Info).

<Tabs items="[&#x22;Docker&#x22;, &#x22;Helm&#x22;]">
  <Tab value="Docker">
    ```yaml title="config.yaml"
    log:
      level: 2   # 0=Trace 1=Debug 2=Info 3=Warn 4=Error 5=Fatal
    ```

    Or set it with an environment variable, which overrides the config-file value at startup:

    ```bash
    LOGLEVEL=1   # Debug
    ```
  </Tab>

  <Tab value="Helm">
    ```yaml title="values.yaml"
    log:
      level: 2   # 0=Trace 1=Debug 2=Info 3=Warn 4=Error 5=Fatal
    ```
  </Tab>
</Tabs>

## Change the level at runtime [#change-the-level-at-runtime]

The log level can be changed **without restarting the server**. The management API exposes a
`set_log_level` action on its unified request endpoint; a single call takes effect immediately
across every component, because all loggers share one level. See the
[Action Endpoint](/operate/observability/api-reference/actions) for how requests are sent to the
management API.

## Correlate logs with traces [#correlate-logs-with-traces]

When [OpenTelemetry tracing](/operate/observability/tracing) is enabled, log lines emitted during a
traced operation carry `trace_id` and `span_id` fields. This lets you pivot from a log line to
the matching trace (and back) in your backend — Jaeger, Grafana Tempo, Datadog, and others.

```json
{
  "level": "ERROR",
  "time": "2026-06-29T12:34:56.789Z",
  "component": "array",
  "caller": "array/events_sender.go:42",
  "msg": "publish pub/sub event error",
  "host": "kubemq-node-1",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "error": "context deadline exceeded"
}
```

These two fields appear **only when tracing is on and a span is active** for the operation being
logged. With tracing disabled, the fields are simply absent and logging behaves exactly as
before. Correlation therefore requires OpenTelemetry — see the
[Distributed Tracing](/operate/observability/tracing) page to enable it.

## Ship logs [#ship-logs]

Because KubeMQ logs JSON to stdout, any log shipper that captures container stdout works without
extra configuration — **Loki**, **Splunk**, **Datadog**, and **Fluentd** all ingest the lines as
structured records.

To correlate logs and traces in Grafana, link a Loki datasource to a Tempo datasource:

1. Ship KubeMQ's JSON logs to Loki (for example via Promtail or the Grafana Agent).
2. In the Loki datasource settings, configure **Derived Fields** with a regex that extracts
   `trace_id`, pointing it at the Tempo datasource.
3. Log lines from traced operations then show a clickable **Tempo** button that opens the trace
   in the same time range.

For Datadog, the `trace_id` and `span_id` field names already match the fields Datadog expects
for log–trace correlation, once your log pipeline parses them from the JSON body.

## Related [#related]

<Cards>
  <Card title="Distributed Tracing" href="/operate/observability/tracing" description="Enable OpenTelemetry to populate trace_id and span_id on log lines." />

  <Card title="Action Endpoint" href="/operate/observability/api-reference/actions" description="Change the log level at runtime via the management API." />
</Cards>