Durability & Retention
How Kafka durability maps to KubeMQ — the acks 0/1/all contract, quorum-fsynced zero-acked-loss, retention by time and size, and log compaction.
Kafka clients control durability with one producer setting — acks — and control how long a record lives with two topic settings — retention.ms and retention.bytes — plus an alternative to time-based deletion, log compaction. All three map onto KubeMQ's next storage engine in ways that are mostly Kafka-identical, with a couple of differences worth knowing before you deploy to a multi-node cluster.
Overview
Every Kafka topic on KubeMQ backs onto an Events Store log on the next engine — an owned segment log replicated with Dragonboat raft. The Kafka connector only runs on next (never on the legacy engine), because next is what makes Kafka's headline durability and compaction contracts possible in the first place. A fresh store with the Kafka connector enabled auto-selects next automatically — see Zero-config engine selection for the full selection rules; this page does not repeat them.
The acks contract
A Kafka producer's acks setting is the wire-level durability request attached to every Produce. KubeMQ accepts the three Kafka-legal values — any other value is rejected with INVALID_REQUIRED_ACKS before the record is ever stored:
acks | Behavior | Response |
|---|---|---|
0 | Fire-and-forget. The record is still appended to the store, but the connector never inspects the outcome and returns no response frame — the producer gets no confirmation, successful or not | None |
1 | Leader acknowledgement. If the request lands on a follower node, the follower transparently forwards it to the leader; the producer waits for the write to complete | Response frame with the assigned offset (or an error) |
-1 / all | All in-sync replicas. On KubeMQ today this is answered identically to acks=1 — the in-sync-replica set is exactly the leader itself, so there is no separate follower-ack tier to wait on beyond what next's own quorum commit already provides | Response frame with the assigned offset (or an error) |
Quorum-fsynced durability on the next engine
Once a produce request reaches the leader — either directly, or forwarded there because acks was 1 or all — the record is committed to a raft quorum and fsynced before the connector replies. This is the next engine's durability contract, and it holds regardless of which acks value the client asked for: next never acknowledges a write that has not already cleared quorum-replication and fsync.
Put plainly: an acked message survives a node loss. That is a real, meaningful guarantee — but it is a comparison of defaults, not a marketing claim of "zero data loss" in the absolute sense. It says nothing about a message that was never durably written in the first place, which is precisely what can happen under acks=0 (below). "Zero acked-loss" describes what happens to a message after it is acknowledged, not a promise that every produced byte is retained forever regardless of the client's own settings.
Every produced record is committed to a raft quorum and fsynced on the next engine before the connector responds — the same durability path runs regardless of the requested acks level; what changes is whether the client waits for, and receives, that confirmation.
A multi-node install requires acks >= 1. The default Helm/CRD install runs replicas: 3 fronted by a single Service, so a produce request can land on any pod. For acks >= 1, a follower transparently forwards the request to the leader. For acks=0 (fire-and-forget), a follower silently drops the record instead of forwarding it — the producer gets no error, and the record never reaches the store. Use acks >= 1 on any multi-node deployment; single-node/standalone deployments are unaffected because there is no follower to drop it.
Retention: time and size
A topic's retention is governed by two config keys, and only one of them currently does anything on KubeMQ:
| Config | Default | Enforced? | Behavior |
|---|---|---|---|
retention.ms | 604800000 (7 days) — display only, the value DescribeConfigs echoes for a topic that never set it | Yes, but only once a topic sets a finite, positive value | retention.ms=-1, 0, or unset all pin the topic — records are never age-evicted. Only a finite positive value activates the leader-gated sweep, which runs every 10 seconds and evicts records older than the window by logically advancing the log's start offset. This differs from real Apache Kafka, where 0 evicts almost immediately |
retention.bytes | — | No | Accepted and echoed back if a client sets it, but there is no size-based eviction sweep today — only a finite retention.ms actually removes records |
If you are used to real Kafka enforcing both a time and a size bound, note the gap: on KubeMQ today, a topic that never sets retention.ms keeps growing (the store's log-start never advances), and setting retention.bytes alone does not cap anything. If you need a bound, set retention.ms. Retention eviction never renumbers offsets — an evicted record's slot simply becomes unreadable; surviving records keep the offsets they always had.
This is distinct from offset retention — how long a consumer group's committed offsets are kept once a group has no active members — which is a separate, connector-wide setting covered in Limits & Rules; this page is about record retention on the topic itself, not group-offset retention.
Log compaction
Instead of (or alongside) time-based deletion, a topic can be compacted: KubeMQ keeps only the latest record per key, discarding older values for the same key. Compaction is controlled by cleanup.policy, which accepts:
| Value | Meaning |
|---|---|
delete (default) | Time/size-based retention only, as described above — no compaction |
compact | Latest-value-per-key retention only |
compact,delete | Both — compact by key, and also age out anything older than retention.ms |
A compact-only topic is never age-evicted even if retention.ms is set to a finite value — only compact,delete combines the two. Setting retention.ms on a compact-only topic has no effect until cleanup.policy also includes delete.
A record with a null value and a non-null key is a tombstone — a marker that erases the key. Tombstones are themselves reaped (physically removed) after delete.retention.ms (default 24 hours, 86400000), giving downstream consumers a window to observe the delete before it disappears. Crucially, compaction never renumbers surviving offsets — when a compacted-away record's offset is fetched, the read simply returns the next surviving record at or after it, exactly like real Kafka's own compacted-topic behavior, which every conformant Kafka client already knows how to handle.
Log compaction is a next-engine-only capability: it runs on Kafka topic channels specifically, and since the Kafka connector itself only runs on next, every Kafka topic on KubeMQ is eligible. Native Queues and Events Store channels used by KubeMQ's other patterns never compact — this is a Kafka-topic-specific feature, not a general storage behavior.
Compaction is what unlocks the compaction-dependent ecosystem: Kafka Connect and Kafka Streams both rely on internally compacted topics (offset storage, changelog topics), so having real cleanup.policy=compact support means those tools work against KubeMQ too. Compaction runs live, at any point in a topic's life — turning it on is a runtime cleanup.policy change, not a data-migration step. If you are moving an existing Kafka workload onto KubeMQ, that move is a separate, start-fresh adoption story with its own assess/replicate/cutover playbook — see Migrate from Kafka rather than treating compaction as a migration blocker.
Related
Compacted Topics
Turn on cleanup.policy=compact, produce keyed records and tombstones, and verify latest-value-per-key retention.
Storage Engines: zero-config selection
How a fresh store auto-selects the next engine when Kafka is enabled.
Producing
Set acks, produce keyed and headered records, and use the idempotent producer.
Was this page helpful?
Partitions & Ordering
The Kafka partition model on KubeMQ — 1..256 increase-only partitions, client-side key hashing (murmur2 vs CRC32), and the per-partition ordering guarantee.
Getting Started
Enable the Kafka connector, then produce and consume your first message over the wire protocol with kcat and seven client libraries — a full round-trip.