Durability & Retention
Kafka acknowledgements and strict versus fast storage durability, with retention by time and size and log compaction on KubeMQ.
Kafka clients control durability with one producer setting — acks — and control how long a record lives with two topic settings — retention.ms and retention.bytes — plus an alternative to time-based deletion, log compaction. All three map onto KubeMQ's next storage engine in ways that are mostly Kafka-identical, with a couple of differences worth knowing before you deploy to a multi-node cluster.
Overview
Every Kafka topic on KubeMQ backs onto an Events Store log on the next storage engine — an owned segment log replicated with Dragonboat raft. The Kafka connector only runs on next (never on the legacy engine), because next is what makes Kafka's headline durability and compaction contracts possible in the first place. A fresh store with the Kafka connector enabled auto-selects next automatically — see Zero-config engine selection for the full selection rules; this page does not repeat them.
The acks contract
A Kafka producer's acks setting is the wire-level durability request attached to every Produce. KubeMQ accepts the three Kafka-legal values — any other value is rejected with INVALID_REQUIRED_ACKS before the record is ever stored:
acks | Behavior | Response |
|---|---|---|
0 | Fire-and-forget. The record is still appended to the store, but the connector never inspects the outcome and returns no response frame — the producer gets no confirmation, successful or not | None |
1 | Leader acknowledgement. If the request lands on a follower node, the follower transparently forwards it to the leader; the producer waits for the write to complete | Response frame with the assigned offset (or an error) |
-1 / all | Uses the same storage acknowledgement boundary as acks=1. The broker's selected fast or strict policy governs when its quorum write is acknowledged; the client setting does not force disk synchronization | Response frame with the assigned offset (or an error) |
Quorum-fsynced durability on the next storage engine
The engine acknowledgement policy and the producer's acks setting are separate.
For KubeMQ 3.6.21, STORE_NEXT_ACK_POLICY is read at broker startup and accepts:
| Broker policy | A successful response means | Power-loss boundary |
|---|---|---|
fast (default) | The replicated log entry has reached a quorum and been written through to the operating system | Disk synchronization follows in a bounded background window. Losing power across the quorum before it completes can lose acknowledged writes |
strict | The replicated log entry has been synchronized to disk on a quorum before acknowledgement | Removes that unsynchronized acknowledgement window; retention, loss of the required replicas and storage destruction remain separate concerns |
Use STORE_NEXT_ACK_POLICY=strict when the workload or migration requires a response
to follow quorum disk synchronization. Set it explicitly on the target before starting
the migration. A recipe that chooses strict is stricter than the broker default;
acks=all by itself does not change the engine policy. See
Ack policy for configuration examples.
A standalone server has only one local copy. Quorum replication does not turn a single-node evaluation into a multi-server installation. These acknowledgement boundaries are not an unconditional guarantee against every form of data loss.
The selected engine policy determines the acknowledgement boundary: quorum replication and operating-system write-through under fast, or quorum disk synchronization under strict. The producer's acks setting determines whether it receives a response.
A multi-node install requires acks >= 1. The default Helm/CRD install runs replicas: 3 fronted by a single Service, so a produce request can land on any pod. For acks >= 1, a follower transparently forwards the request to the leader. For acks=0 (fire-and-forget), a follower silently drops the record instead of forwarding it — the producer gets no error, and the record never reaches the store. Use acks >= 1 on any multi-node deployment; single-node/standalone deployments are unaffected because there is no follower to drop it.
Retention: time and size
A topic's retention is governed by two config keys, and only one of them currently does anything on KubeMQ:
| Config | Default | Enforced? | Behavior |
|---|---|---|---|
retention.ms | 604800000 (7 days) — display only, the value DescribeConfigs echoes for a topic that never set it | Yes, but only once a topic sets a finite, positive value | retention.ms=-1, 0, or unset all pin the topic — records are never age-evicted. Only a finite positive value activates the leader-gated sweep, which runs every 10 seconds and evicts records older than the window by logically advancing the log's start offset. This differs from real Apache Kafka, where 0 evicts almost immediately |
retention.bytes | — | No | Accepted and echoed back if a client sets it, but there is no size-based eviction sweep today — only a finite retention.ms actually removes records |
If you are used to real Kafka enforcing both a time and a size bound, note the gap: on KubeMQ today, a topic that never sets retention.ms keeps growing (the store's log-start never advances), and setting retention.bytes alone does not cap anything. If you need a bound, set retention.ms. Retention eviction never renumbers offsets — an evicted record's slot simply becomes unreadable; surviving records keep the offsets they always had.
This is distinct from offset retention — how long a consumer group's committed offsets are kept once a group has no active members — which is a separate, connector-wide setting covered in Limits & Rules; this page is about record retention on the topic itself, not group-offset retention.
Log compaction
Instead of (or alongside) time-based deletion, a topic can be compacted: KubeMQ keeps only the latest record per key, discarding older values for the same key. Compaction is controlled by cleanup.policy, which accepts:
| Value | Meaning |
|---|---|
delete (default) | Time/size-based retention only, as described above — no compaction |
compact | Latest-value-per-key retention only |
compact,delete | Both — compact by key, and also age out anything older than retention.ms |
A compact-only topic is never age-evicted even if retention.ms is set to a finite value — only compact,delete combines the two. Setting retention.ms on a compact-only topic has no effect until cleanup.policy also includes delete.
A record with a null value and a non-null key is a tombstone — a marker that erases the key. Tombstones are themselves reaped (physically removed) after delete.retention.ms (default 24 hours, 86400000), giving downstream consumers a window to observe the delete before it disappears. Crucially, compaction never renumbers surviving offsets — when a compacted-away record's offset is fetched, the read simply returns the next surviving record at or after it, exactly like real Kafka's own compacted-topic behavior, which every conformant Kafka client already knows how to handle.
Log compaction is a next-engine-only capability: it runs on Kafka topic channels specifically, and since the Kafka connector itself only runs on next, every Kafka topic on KubeMQ is eligible. Native Queues and Events Store channels used by KubeMQ's other patterns never compact — this is a Kafka-topic-specific feature, not a general storage behavior.
Compaction is what unlocks the compaction-dependent ecosystem: Kafka Connect and Kafka Streams both rely on internally compacted topics (offset storage, changelog topics), so having real cleanup.policy=compact support means those tools work against KubeMQ too. Compaction runs live, at any point in a topic's life — turning it on is a runtime cleanup.policy change, not a data-migration step. If you are moving an existing Kafka workload onto KubeMQ, compaction is not a blocker: kmq migrate copies topics — compacted ones included — records and consumer-group offsets through its assess/replicate/translate/cutover phases — see Migrate from Kafka.
Related
Compacted Topics
Turn on cleanup.policy=compact, produce keyed records and tombstones, and verify latest-value-per-key retention.
Storage Engines: zero-config selection
How a fresh store auto-selects the next storage engine when Kafka is enabled.
Producing
Set acks, produce keyed and headered records, and use the idempotent producer.
Was this page helpful?
Partitions & Ordering
The Kafka partition model on KubeMQ — 1..256 increase-only partitions, client-side key hashing (murmur2 vs CRC32), and the per-partition ordering guarantee.
What Differs from Kafka
Understand the KubeMQ Kafka behaviors that need attention before moving a production workload from Apache Kafka, including share groups.