# Durability & Retention (/connectors/kafka/concepts/durability-and-retention)
Kafka clients control durability with one producer setting — `acks` — and control how long a record lives with two topic settings — `retention.ms` and `retention.bytes` — plus an alternative to time-based deletion, log compaction. All three map onto KubeMQ's `next` storage engine in ways that are mostly Kafka-identical, with a couple of differences worth knowing before you deploy to a multi-node cluster.
## Overview [#overview]
Every Kafka topic on KubeMQ backs onto an Events Store log on the `next` engine — an owned segment log replicated with Dragonboat raft. The Kafka connector only runs on `next` (never on the legacy engine), because `next` is what makes Kafka's headline durability and compaction contracts possible in the first place. A fresh store with the Kafka connector enabled auto-selects `next` automatically — see [Zero-config engine selection](/configure/reference/storage-engines#zero-config-engine-selection) for the full selection rules; this page does not repeat them.
## The `acks` contract [#the-acks-contract]
A Kafka producer's `acks` setting is the wire-level durability request attached to every `Produce`. KubeMQ accepts the three Kafka-legal values — any other value is rejected with `INVALID_REQUIRED_ACKS` before the record is ever stored:
| `acks` | Behavior | Response |
| ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
| `0` | Fire-and-forget. The record is still appended to the store, but the connector never inspects the outcome and returns no response frame — the producer gets no confirmation, successful or not | None |
| `1` | Leader acknowledgement. If the request lands on a follower node, the follower transparently forwards it to the leader; the producer waits for the write to complete | Response frame with the assigned offset (or an error) |
| `-1` / `all` | All in-sync replicas. On KubeMQ today this is answered identically to `acks=1` — the in-sync-replica set is exactly the leader itself, so there is no separate follower-ack tier to wait on beyond what `next`'s own quorum commit already provides | Response frame with the assigned offset (or an error) |
## Quorum-fsynced durability on the `next` engine [#quorum-fsynced-durability-on-the-next-engine]
Once a produce request reaches the leader — either directly, or forwarded there because `acks` was 1 or all — the record is committed to a **raft quorum and fsynced** before the connector replies. This is the `next` engine's durability contract, and it holds regardless of which `acks` value the client asked for: `next` never acknowledges a write that has not already cleared quorum-replication and fsync.
Put plainly: **an acked message survives a node loss.** That is a real, meaningful guarantee — but it is a comparison of defaults, not a marketing claim of "zero data loss" in the absolute sense. It says nothing about a message that was never durably written in the first place, which is precisely what can happen under `acks=0` (below). "Zero acked-loss" describes what happens to a message *after* it is acknowledged, not a promise that every produced byte is retained forever regardless of the client's own settings.
*Every produced record is committed to a raft quorum and fsynced on the `next` engine before the connector responds — the same durability path runs regardless of the requested `acks` level; what changes is whether the client waits for, and receives, that confirmation.*
**A multi-node install requires `acks >= 1`.** The default Helm/CRD install runs `replicas: 3` fronted by a single Service, so a produce request can land on any pod. For `acks >= 1`, a follower transparently forwards the request to the leader. For `acks=0` (fire-and-forget), a follower **silently drops** the record instead of forwarding it — the producer gets no error, and the record never reaches the store. Use `acks >= 1` on any multi-node deployment; single-node/standalone deployments are unaffected because there is no follower to drop it.
## Retention: time and size [#retention-time-and-size]
A topic's retention is governed by two config keys, and only one of them currently does anything on KubeMQ:
| Config | Default | Enforced? | Behavior |
| ----------------- | --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `retention.ms` | `604800000` (7 days) — **display only**, the value `DescribeConfigs` echoes for a topic that never set it | **Yes, but only once a topic sets a finite, positive value** | `retention.ms=-1`, `0`, or **unset** all pin the topic — records are never age-evicted. Only a finite positive value activates the leader-gated sweep, which runs every 10 seconds and evicts records older than the window by logically advancing the log's start offset. This differs from real Apache Kafka, where `0` evicts almost immediately |
| `retention.bytes` | — | **No** | Accepted and echoed back if a client sets it, but there is no size-based eviction sweep today — only a finite `retention.ms` actually removes records |
If you are used to real Kafka enforcing both a time and a size bound, note the gap: on KubeMQ today, a topic that never sets `retention.ms` keeps growing (the store's log-start never advances), and setting `retention.bytes` alone does not cap anything. If you need a bound, set `retention.ms`. Retention eviction never renumbers offsets — an evicted record's slot simply becomes unreadable; surviving records keep the offsets they always had.
This is distinct from **offset retention** — how long a consumer group's committed offsets are kept once a group has no active members — which is a separate, connector-wide setting covered in [Limits & Rules](/connectors/kafka/reference/limits-and-rules); this page is about record retention on the topic itself, not group-offset retention.
## Log compaction [#log-compaction]
Instead of (or alongside) time-based deletion, a topic can be compacted: KubeMQ keeps only the **latest record per key**, discarding older values for the same key. Compaction is controlled by `cleanup.policy`, which accepts:
| Value | Meaning |
| ------------------ | -------------------------------------------------------------------------- |
| `delete` (default) | Time/size-based retention only, as described above — no compaction |
| `compact` | Latest-value-per-key retention only |
| `compact,delete` | Both — compact by key, and also age out anything older than `retention.ms` |
A **`compact`-only** topic is never age-evicted even if `retention.ms` is set to a finite value — only `compact,delete` combines the two. Setting `retention.ms` on a `compact`-only topic has no effect until `cleanup.policy` also includes `delete`.
A record with a **null value** and a non-null key is a **tombstone** — a marker that erases the key. Tombstones are themselves reaped (physically removed) after `delete.retention.ms` (default 24 hours, `86400000`), giving downstream consumers a window to observe the delete before it disappears. Crucially, **compaction never renumbers surviving offsets** — when a compacted-away record's offset is fetched, the read simply returns the next surviving record at or after it, exactly like real Kafka's own compacted-topic behavior, which every conformant Kafka client already knows how to handle.
Log compaction is a **`next`-engine-only** capability: it runs on Kafka topic channels specifically, and since the Kafka connector itself only runs on `next`, every Kafka topic on KubeMQ is eligible. Native Queues and Events Store channels used by KubeMQ's other patterns never compact — this is a Kafka-topic-specific feature, not a general storage behavior.
Compaction is what unlocks the compaction-dependent ecosystem: **Kafka Connect** and **Kafka Streams** both rely on internally compacted topics (offset storage, changelog topics), so having real `cleanup.policy=compact` support means those tools work against KubeMQ too. Compaction runs live, at any point in a topic's life — turning it on is a runtime `cleanup.policy` change, not a data-migration step. If you are moving an existing Kafka workload onto KubeMQ, that move is a separate, start-fresh adoption story with its own assess/replicate/cutover playbook — see [Migrate from Kafka](/connectors/kafka/how-to/migrate-from-kafka) rather than treating compaction as a migration blocker.
## Related [#related]