# Storage Engines (/configure/reference/storage-engines)
KubeMQ ships **two persistence engines**: `next` and `legacy`. The engine
governs **only the persistence plane** — the Events Store and Queues. Ephemeral patterns
(Events pub/sub, Commands/Queries RPC) ride the same internal messaging core in both modes
and behave **identically** regardless of which engine a cluster runs.
## The two engines [#the-two-engines]
| | `legacy` | `next` (resolved on a clean store) |
| ---------------------------------------------- | ------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Persistence | the legacy persistence engine (file store) | owned segment log + Dragonboat raft |
| Durability on ack | accepted into the legacy file store; fsync on a timer (`Broker.DiskSyncSeconds`, default 5s) | **selectable.** `fast` (**the default**): ack after quorum-replication, fsync follows in a bounded background window. `strict`: ack only after quorum-replication **and** fsync (zero acked-loss). See [Ack policy](#ack-policy) |
| Cluster consensus | file-store raft over the cluster mesh | Dragonboat raft over a dedicated replication listener (optional mTLS, default off) |
| Kafka compaction | not supported (Kafka is `next`-only) | supported — unlocks Kafka Connect and Kafka Streams |
| Native store limits & retention (`store.max*`) | enforced | **not consumed** — native channels have no age/size/count cap ([details](#native-retention-scope)) |
| Status | **feature-frozen** (maintenance/security/bug fixes only); opt-in — name it explicitly at creation | what a clean store resolves to; receives all new durability and compaction features |
## Choosing an engine [#choosing-an-engine]
* **A new deployment on a clean store comes up on `next`.** There is nothing to set — see
[Zero-config engine selection](#zero-config-engine-selection) below.
* **`next` is also required for Kafka** compacted topics, and it is the only engine that
can give you quorum-fsynced durability — **but not by default**: that needs
`store.nextackpolicy: strict`. See [Ack policy](#ack-policy).
* **To run `legacy`, name it explicitly at creation** — `store.engine: legacy` (Docker) or
`spec.store.engine: legacy` (Kubernetes). It is no longer what you get by omission.
* **An existing cluster keeps the engine it was born with.** There is no migration path
between engines, and nothing about the change above rolls or reinterprets a running
cluster.
**If you have a page, script, or values file that assumes `legacy` is what you get by
default, it is now wrong.** A fresh cluster with a clean store resolves to `next`. This
changed in operator **v2.3.0** on Kubernetes; on Docker the server has always resolved a
clean store to `next` when the engine was left unset.
## Zero-config engine selection [#zero-config-engine-selection]
An explicit engine always wins and skips every probe below: `store.engine` (Docker
`config.yaml`), `STORE_ENGINE` (env), or `spec.store.engine` (Kubernetes).
Otherwise **the server resolves the engine itself at boot**, by probing the store
directory:
* **The directory already holds a store** → the engine that wrote it.
* **The directory is clean** → `next`.
The resolved engine is named in a **boot NOTICE** in the server log, along with the
deciding signal and the directory that was probed. When the choice was delegated rather
than pinned, **that log line is the only place the running engine is reported** — see
[Which engine am I actually on?](#which-engine-am-i-actually-on) below.
### On Kubernetes the operator delegates, with `STORE_ENGINE=auto` [#on-kubernetes-the-operator-delegates-with-store_engineauto]
From operator **v2.3.0**, a cluster with **no engine on record** is created with
`STORE_ENGINE=auto` and the server resolves it as above. Established clusters are
untouched — their engine stays named explicitly and nothing rolls.
Three consequences worth holding onto:
* **`status.engine` and the `core.k8s.kubemq.io/established-engine` annotation both read
`auto`** on such a cluster. That value records *the delegation*, not the engine running.
* **Setting `spec.store.engine` later on an `auto` cluster is allowed.** `auto` records
that the cluster delegated the choice, not an established engine, so pinning afterwards
is a first choice rather than an engine change. **Pin the engine the server actually
resolved** — a pin that disagrees with the data on disk is refused by the server at
boot, and the server never deletes a datadir.
* **On an established cluster the annotation is authoritative and outranks
`spec.store.engine`.** A mismatch is refused, not silently applied. See the
[engine-establishment guard](/configure/reference/deployment#engine-establishment-guard).
### Which engine am I actually on? [#which-engine-am-i-actually-on]
| Cluster | Where to read the engine |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Engine named explicitly | `spec.store.engine`, `status.engine`, or the annotation — all agree |
| Engine delegated (`auto`) | **the server's boot NOTICE in the pod log**, or the two API fields below. `status.engine` and the annotation both say `auto`, which is the delegation, not the answer |
```bash
kubectl logs -n | grep -i 'store engine'
```
The management API's config view carries the same answer in two read-only fields — they
exist precisely because this question was previously unanswerable on a delegating
deployment. Neither is an operator input, and neither is ever written back into a saved
config file:
| Field | What it holds |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `engineResolved` | The engine this node is **actually running** — always a concrete `legacy` or `next`, never `auto`. |
| `engineResolvedSignal` | **Why** it holds that, in plain words: `clean store (auto default)`, `explicitly configured (detection did not run)`, `clean store (CleanStore=true)`, or the on-disk evidence the probe matched. |
The signal matters as much as the engine. A node reporting `legacy` may have been pinned by
an operator, or may have detected legacy data on disk — and the remedy differs.
### Kafka [#kafka]
Kafka runs on `next` only, and needs no manual engine step:
* **Clean store + Kafka enabled + engine unset** → resolves to `next` (NOTICE logged) —
the same clean-store outcome as without Kafka.
* **Existing `legacy` store data + Kafka enabled** → fails closed with a config error
naming the conflicting store directory. Kafka never ran on `legacy`, so the server
refuses to mix engines rather than silently reinterpreting existing data.
* **Explicit `store.engine: legacy` + Kafka enabled** → rejected at boot. Kafka requires
the `next` engine; naming `legacy` alongside it is a configuration error.
* **Naming `next` explicitly** skips the probe entirely and always wins — the predictable
choice for IaC/GitOps that shouldn't depend on probe-time filesystem state.
## Mode isolation [#mode-isolation]
A cluster is **born one mode and stays there**. The data directory records a mode marker
at first boot. Every later boot compares that marker against the configured engine — a
**mismatch is a fatal boot error**: the server refuses to start, and it **never wipes the
directory**. There is no in-place engine migration and no cross-format compatibility
between `legacy` and `next` data.
## Durability guarantees [#durability-guarantees]
Three distinct dimensions govern durability, and they should not be conflated:
1. **Ack meaning.** On `legacy`, a publish is acknowledged after the write lands in the
file store; the store is fsynced on a timer (`Broker.DiskSyncSeconds`, default 5s) —
so an ack can precede the fsync. On `next`, **it depends on the ack policy** below:
under `strict` a publish is acknowledged only after the write is quorum-replicated
**and** fsynced; under `fast` — the shipped default — it is acknowledged once
quorum-replicated, with the fsync following in a bounded background window.
2. **Loss window.** "Zero acked-loss" describes **`next` with `strict` only**. It means an
**acked** message is guaranteed to survive. Under the default `fast`, an ack does
**not** imply the record is on disk: writes acknowledged inside the flush window can be
lost to power loss on a whole-quorum failure. And on any engine or policy, "zero
acked-loss" never means zero loss overall — unacked, in-flight work can still be lost
if the client never receives the ack.
3. **Consistency under failover.** On `next`, per-channel sequence numbers are assigned
from the committed raft apply order — they are **gap-free, monotonic, and
restart-stable**, even across a leader election.
## Next-engine settings [#next-engine-settings]
Six `next`-engine settings live on the store config. All are `config.yaml` / env-only —
none has a typed CRD field, so on Kubernetes they travel through `spec.env` or a mounted
config. Every one of them parses and validates on **both** engines; they are simply inert
under `legacy`, so a config that would be wrong after an engine switch fails when it is
written rather than when it becomes load-bearing.
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
| --------------------- | ------------- | ----------------------------- | ------------------ | --------------------------------------------------------- | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Ack policy | string (enum) | `fast` | `fast` \| `strict` | `store.nextackpolicy` · `STORE_NEXT_ACK_POLICY` | — | The ack durability contract — see [Ack policy](#ack-policy) below. **Exact match only**: `Fast`, `STRICT` and `true` are all rejected. |
| Raft address | string | `""` → `127.0.0.1:` | `host:port` | `store.nextraftaddress` · `STORE_NEXT_RAFT_ADDRESS` | — | The engine's Dragonboat replication-listener bind address. Empty computes a single-node loopback default. A standalone node pins the listener to loopback regardless — it must not be routable. |
| Segment size (bytes) | int64 | `0` → engine default (64 MiB) | ≥ `0` | `store.nextsegmentsize` · `STORE_NEXT_SEGMENT_SIZE` | — | Per-channel segment-log size, the analog of Kafka's `segment.bytes`. A smaller value rolls segments sooner, which is what lets compaction reclaim space promptly on low-volume topics. |
| Shard-pool size | int | `1` | `1`–`16` | `store.nextshards` · `STORE_NEXT_SHARDS` | — | Number of independent raft groups the persistent data plane is partitioned across, with channels hash-mapped onto them. `1` is the GA layout and is behavior-identical to it. `0` normalizes to `1`; above `16` is rejected. |
| Balance shard leaders | bool | `false` | true / false | `store.nextbalanceleaders` · `STORE_NEXT_BALANCE_LEADERS` | — | Opt-in: spreads shard leaders across cluster nodes instead of pinning them all to one. Meaningless without a shard pool > 1 — rejected unless the cluster is clustered `next` with `nextshards` > 1. `false` is the shipped GA behavior. |
### Ack policy [#ack-policy]
**The shipped default is `fast`, and under `fast` an ack does NOT mean the record is on
disk.** A fresh `next` cluster does not give you the quorum-fsynced ack unless you ask for
it.
| | `fast` (**default**) | `strict` |
| ----------- | --------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
| Ack means | the entry is **quorum-replicated**; the raft log fsync follows in the background, within a bounded window | the entry is quorum-replicated **and fsynced on a quorum** of raft logs |
| Loss window | writes acked inside the flush window can be lost to a whole-quorum power loss | none — this is the "zero acked-loss" contract |
| Cost | lower publish latency | fsync on the publish path |
If a compliance requirement or an audit answer depends on *"an ack means it is on disk"*,
set it explicitly:
```yaml title="config.yaml"
store:
nextackpolicy: strict
```
On Kubernetes, via `spec.env`:
```yaml title="values.yaml"
env:
STORE_NEXT_ACK_POLICY: "strict"
```
## Native retention scope [#native-retention-scope]
The store-level limit and retention settings — **Max channels**, **Max channel size**,
**Max messages**, **Message retention**, and **Purge inactive** (see
[Storage & Queues](/configure/reference/storage-queues)) — are enforced by the
**`legacy` engine only**. The `next` engine does not consume them: a native Events Store
or Queues channel on `next` has **no age, size, or count cap** and grows unbounded under
a slow or absent consumer. The queue ack-wait timeout governs redelivery of in-flight
messages — it is not a backlog-eviction bound.
On `next`, the only age-evicted channels are **Kafka topic channels**, via the Kafka
connector's `retention.ms` (see
[Durability & Retention](/connectors/kafka/concepts/durability-and-retention#retention-time-and-size)).
Size native Events Store / Queues workloads on `next` by disk capacity, or use Kafka
topic channels where age eviction matters.
**This now applies to new clusters by default.** Because a clean store resolves to `next`,
a freshly installed cluster has **no age, size, or count cap** on native Events Store and
Queues channels unless you set one — the `store.max*` limits are `legacy`-only. Size by
disk, and set a `spec.volume.size` you can live with.
## Support & deprecation [#support--deprecation]
Both engines are **fully supported today**. `legacy` is **feature-frozen** — it receives
maintenance, security, and bug fixes only. `next` receives all new durability and
compaction features going forward. The deprecation policy for `legacy` is **explicitly
TBD** — no sunset date has been set, and both engines are supported until one is.
## Compaction [#compaction]
Kafka log compaction is **`next`-only**. The Kafka connector's `cleanup.policy` accepts
`delete` (default), `compact`, or `compact,delete`. Compaction keeps only the latest
record per key and reaps tombstones after `delete.retention.ms` (default 24h) — it never
renumbers surviving offsets. Compaction is scoped to Kafka topic channels only; native
Queues and Events Store channels never compact. This is the feature that unlocks Kafka
Connect (its internal topics require compaction) and Kafka Streams changelogs.
## Clustered next-engine replication [#clustered-next-engine-replication]
The `Cluster.Replication.*` block configures the `next` engine's Dragonboat replication
listener — a second membership plane a `next`-mode cluster runs alongside the cluster
mesh. It is meaningful **only** when `Store.Engine` is `next` **and** `Cluster.Enable` is
`true`. This block is Docker / `config.yaml` and env-only — there is **no Helm/CRD path**.
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
| -------------------- | ------ | ------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Replica ID | uint64 | `0` | ≥ 0 | `cluster.replication.replicaid` · `CLUSTER_REPLICATION_REPLICA_ID` | — | This node's Dragonboat replica id (1..N); must be a key in `Peers`. May be left `0` when `POD_NAME` is present — see [ReplicaID auto-derive](#replicaid-auto-derive). |
| Raft address | string | `""` | `host:port` | `cluster.replication.raftaddress` · `CLUSTER_REPLICATION_RAFT_ADDRESS` | — | This node's **advertised** raft address. When set it wins over the address derived from `Peers` for what this node announces to its peers — set it when the address peers must dial differs from the one this node binds. |
| Peers | string | `""` | `id@host:port,...` | `cluster.replication.peers` · `CLUSTER_REPLICATION_PEERS` | — | Full initial-members map — every initial member, including self. Byte-identical on every pod. |
| Join | bool | `false` | true / false | `cluster.replication.join` · `CLUSTER_REPLICATION_JOIN` | — | Boot as a later-added replica, with an empty members map. |
| RTT (ms) | uint64 | `200` | ≥ 0 | `cluster.replication.rttmillisecond` · `CLUSTER_REPLICATION_RTT_MILLISECOND` | — | Dragonboat logical-clock tick, in milliseconds. |
| Election RTT | uint64 | `10` | ≥ 0 | `cluster.replication.electionrtt` · `CLUSTER_REPLICATION_ELECTION_RTT` | — | Must be greater than `2 × HeartbeatRTT`. |
| Heartbeat RTT | uint64 | `1` | ≥ 0 | `cluster.replication.heartbeatrtt` · `CLUSTER_REPLICATION_HEARTBEAT_RTT` | — | |
| Snapshot entries | uint64 | `10000` | ≥ 0 | `cluster.replication.snapshotentries` · `CLUSTER_REPLICATION_SNAPSHOT_ENTRIES` | — | Committed entries between automatic snapshots. |
| Compaction overhead | uint64 | `2000` | ≥ 0 | `cluster.replication.compactionoverhead` · `CLUSTER_REPLICATION_COMPACTION_OVERHEAD` | — | Log entries retained past a snapshot. |
| Boot timeout (s) | int | `60` | ≥ 0 | `cluster.replication.boottimeoutseconds` · `CLUSTER_REPLICATION_BOOT_TIMEOUT_SECONDS` | — | Clustered-boot readiness-probe budget, in seconds. |
| Mutual TLS | bool | `false` | true / false | `cluster.replication.mutualtls` · `CLUSTER_REPLICATION_MUTUAL_TLS` | — | Optional mTLS trust domain. Default **off** (plaintext) — see the security callout below. |
| CA / Cert / Key file | string | `""` | file path | `cluster.replication.cafile` / `cluster.replication.certfile` / `cluster.replication.keyfile` · `CLUSTER_REPLICATION_CA_FILE` / `CLUSTER_REPLICATION_CERT_FILE` / `CLUSTER_REPLICATION_KEY_FILE` | — | A **dedicated** trust domain — not reused from `Security`. Required (fail-closed) when `Mutual TLS` is `true`. |
When **Mutual TLS** is left at its default `false`, a clustered `next` node's Dragonboat
replication listener runs an **unauthenticated, FSM-writing raft port**. This is
acceptable only on a trusted pod network, bounded by a `NetworkPolicy` — never expose this
listener beyond the cluster's own pod network.
## ReplicaID auto-derive [#replicaid-auto-derive]
On a `next`-engine cluster with clustering enabled, `Peers` set, `Replica ID` left at `0`,
and `POD_NAME` present in the environment, the server derives
`ReplicaID = ordinal(POD_NAME) + 1` — the integer after the last `-` in the pod name, plus
one (StatefulSet ordinals are 0-based, so replica ids run `1..N`).
* An **explicit non-zero** `Replica ID` always wins — the derive path is skipped.
* An **unparseable** `POD_NAME` (no `-` suffix) fails closed.
* The operator obligation: the host at id `ordinal + 1` in `Peers` must be the DNS name
of the pod at that ordinal — a mismatch is a DNS-time failure (the cluster boots but
never forms quorum), not a config-validation error.
This lets `Peers` be byte-identical across every pod in a StatefulSet, with no per-pod
templating required.
## Kubernetes [#kubernetes]
On Kubernetes, the persistence engine is established once, at cluster creation, by the
operator's engine-establishment guard — it is never changed for a live cluster. See
[Deployment & High Availability](/configure/reference/deployment) for the operator-side
behavior.
## Example [#example]
Set the persistence engine on each target. This is a single-setting snippet — see the
[Docker guide](/configure/docker) and the
[Kubernetes guide](/configure/kubernetes) for complete, runnable configurations.
```yaml title="config.yaml"
store:
engine: next
```
```yaml title="values.yaml"
store:
engine: next
```
## See Also [#see-also]