Structured Logging

KubeMQ writes structured JSON logs to stdout. Every log line is a single JSON object, which makes the output easy to capture and parse with any container log aggregator — Loki, Splunk, Datadog, Fluentd, and others — with no extra log files or rotation to manage.

Log format

Every log line is one JSON object on stdout:

{
  "level": "INFO",
  "time": "2026-06-29T12:34:56.789Z",
  "component": "grpc",
  "caller": "grpc/grpc.go:144",
  "msg": "starting gRPC server",
  "host": "kubemq-node-1"
}

The field set:

Components enrich their loggers with extra context fields (such as client_id or channel) that appear as additional key/value pairs on the relevant lines.

Log levels

KubeMQ maps six application levels onto the JSON level field. Info is the default.

Trace and Debug both emit at the DEBUG JSON level. The numeric values used in configuration are 0=Trace, 1=Debug, 2=Info, 3=Warn, 4=Error, 5=Fatal.

Configure the level

Set the log level at startup with the log.level config key (or its environment-variable binding). The default is 2 (Info).

log:
  level: 2   # 0=Trace 1=Debug 2=Info 3=Warn 4=Error 5=Fatal

LOGLEVEL=1   # Debug

log:
  level: 2   # 0=Trace 1=Debug 2=Info 3=Warn 4=Error 5=Fatal

Change the level at runtime

The log level can be changed without restarting the server. The management API exposes a set_log_level action on its unified request endpoint; a single call takes effect immediately across every component, because all loggers share one level. See the Action Endpoint for how requests are sent to the management API.

Correlate logs with traces

When OpenTelemetry tracing is enabled, log lines emitted during a traced operation carry trace_id and span_id fields. This lets you pivot from a log line to the matching trace (and back) in your backend — Jaeger, Grafana Tempo, Datadog, and others.

{
  "level": "ERROR",
  "time": "2026-06-29T12:34:56.789Z",
  "component": "array",
  "caller": "array/events_sender.go:42",
  "msg": "publish pub/sub event error",
  "host": "kubemq-node-1",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "error": "context deadline exceeded"
}

These two fields appear only when tracing is on and a span is active for the operation being logged. With tracing disabled, the fields are simply absent and logging behaves exactly as before. Correlation therefore requires OpenTelemetry — see the Distributed Tracing page to enable it.

Ship logs

Because KubeMQ logs JSON to stdout, any log shipper that captures container stdout works without extra configuration — Loki, Splunk, Datadog, and Fluentd all ingest the lines as structured records.

To correlate logs and traces in Grafana, link a Loki datasource to a Tempo datasource:

1. Ship KubeMQ's JSON logs to Loki (for example via Promtail or the Grafana Agent).
2. In the Loki datasource settings, configure Derived Fields with a regex that extracts trace_id, pointing it at the Tempo datasource.
3. Log lines from traced operations then show a clickable Tempo button that opens the trace in the same time range.

For Datadog, the trace_id and span_id field names already match the fields Datadog expects for log–trace correlation, once your log pipeline parses them from the JSON body.

Related