Partitions & Ordering

Every Kafka topic on KubeMQ is split into one or more partitions — independent, ordered logs that parallelize produce and consume. A topic's partition count starts at 1, only ever grows, and is hard-capped at 256. Partition assignment — which partition a keyed record lands in — is decided entirely by the client, never by KubeMQ, which makes mixing client libraries on the same keyed topic a real footgun worth understanding before you rely on per-key ordering.

Overview

A topic with N partitions is N independent ordered logs, each backed by its own Events Store channel (kafka.<topic>~<partition> for partition ≥ 1; partition 0 lives at kafka.<topic>). Producers and consumers parallelize across those N logs, but KubeMQ's ordering guarantee — like real Kafka's — only ever applies within a single partition. There is no cross-partition ordering, by design: that is exactly the trade KubeMQ's Kafka connector makes to let many producers and many consumers work a topic concurrently.

Partition count: 1 to 256, increase-only

A newly created topic defaults to a single partition (NumPartitions omitted, or set below 1, is treated as 1). From there, the only way to change the partition count is CreatePartitions — and it is deliberately increase-only:

The 256 ceiling is a hard, non-configurable constant in the connector — there is no setting that raises it. A ValidateOnly dry-run is honored for all of the above: it reports the same accept/reject outcome without making a durable change.

This increase-only design is deliberate, not a missing feature. KubeMQ never auto-grows a topic's partition count in the background — every increase is an explicit, operator-triggered CreatePartitions call, so a partition-count change is always a visible event rather than a silent background re-shard. See Limits & Rules for the full numeric ceiling table and Topic Mapping for exactly how partitions map onto Events Store channels.

Client-side key hashing

KubeMQ never hashes a record key to choose a partition. Every partition assignment for a keyed produce is decided client-side and relayed to KubeMQ opaquely — there is no server-side hashing code in the connector, ever. If a client sets an explicit partition (or uses a manual partitioner), that partition is honored verbatim; if it leaves partitioning to the library's default, the library's own hash(key) % N decides.

This matches real Apache Kafka's own architecture, but it means the partition a key lands in depends on which client library produced it — different libraries ship different default partitioners:

Java and franz-go agree with each other out of the box (same hash family), but a librdkafka-based client hashes the same key to a different partition than a Java or franz-go client would, on the same topic, with the same N.

The same key, hashed by two different client libraries' default partitioners, can land on two different partitions of the same topic — KubeMQ relays the client's partition choice opaquely and never re-hashes it.

The per-partition ordering guarantee

A single partition is one ordered writer: every record a producer sends to that partition is appended in the order it arrived, and a consumer reading that partition in isolation sees records in exactly that order. A single-partition topic is therefore totally ordered — every record, across every producer, in one sequence. The moment a topic has N > 1 partitions, ordering is guaranteed only within each partition; there is no ordering guarantee across partitions, and a consumer that needs a total order across the whole topic must read a single partition (or reconcile order itself using a per-key sequence in the record).

Partitions are also KubeMQ's unit of consume parallelism: a consumer group can have at most one active consumer per partition at a time (see Consumer Groups), so growing N is how you add parallel consume capacity — at the cost of the ordering trade above.

Growing a topic re-shards keys

Because partition assignment is hash(key) % N, changing N changes the modulus every keyed partitioner uses — client-side murmur2, CRC32, or any custom scheme. A key that mapped to partition 1 under N=3 can map to a completely different partition once the topic grows to N=6. Concretely: if all of a key's records went to partition 1 while N=3, the very next record for that same key, produced after a successful increase to N=6, might land on partition 4 instead — a consumer reading only partition 1 will not see the two batches as one continuous ordered stream.

The practical takeaway: a topic's per-key ordering guarantee holds only within one fixed-partition-count epoch — from one partition count to the next CreatePartitions increase. This is exactly why CreatePartitions is increase-only and never automatic: an operator-triggered increase is a visible, deliberate boundary between ordering epochs, not something that happens silently underneath a running producer. If a topic's consumers depend on strict cross-epoch per-key order, either avoid growing N on that topic, or design the consumer to reconcile order at the application layer.

One related, self-healing edge case: right after a leader-side partition-count increase in a clustered deployment, a follower node that has not yet caught up on replication can briefly answer Metadata with a stale, lower partition count than the true value. This mirrors real Apache Kafka's own controller-to-broker propagation window — it never advertises a count larger than the true one, so a client is never misdirected to a partition that does not exist, and the client's next metadata refresh self-heals it.

Related