Topic Mapping

This is the master reference for how the embedded KubeMQ Kafka connector maps Kafka topics, partitions, and offsets onto KubeMQ channels. Every topic is backed by one or more KubeMQ Events Store logs — one per partition — plus a small family of internal, reserved channels that hold consumer-group offsets, per-topic config, and transaction/producer state. This deterministic mapping is what lets a native gRPC/REST Events Store subscriber read the exact same data a Kafka client produced, on the same cluster, with no translation step.

Topic grammar

A Kafka topic maps to one KubeMQ Events Store log per partition. Partition 0 gets the bare, back-compat form; every additional partition appends a reserved separator and its index:

kafka.{topic}
└──┬─┘└──┬──┘
   │     └─ the bare topic id (the name a client passes to Produce/Fetch/CreateTopics)
   └─ fixed connector prefix ("kafka."), partition 0 only

kafka.{topic}~{partition}
└──┬─┘└──┬──┘└┬┘└───┬────┘
   │     │    │     └─ the partition index (1, 2, 3, …)
   │     │    └─ the reserved "~" partition separator
   │     └─ the bare topic id
   └─ fixed connector prefix ("kafka."), partition ≥ 1

A topic starts at 1 partition (CreateTopics's NumPartitions, or the implicit single-partition default on auto-create) and can only ever grow — see Partitions & Ordering for the increase-only mechanism and the per-key ordering guarantees that come with it. Each additional partition synthesizes as its own independent channel, with its own ordered offset space, its own log-start, and its own retention/compaction state.

Offsets are Events Store sequence numbers

The connector keeps no separate offset index. A partition's Kafka offsets and its channel's Events Store Sequence numbers are the same counter, one apart: KubeMQ numbers Sequence starting at 1, Kafka numbers offsets starting at 0, so for every record:

offset = Sequence − 1        (Sequence = offset + 1)

That single fact is what makes a robust drop-in possible without KubeMQ maintaining a shadow index. Because Sequence is durable, restart-stable, and identical across every node of a Raft cluster, a Kafka offset inherits all three properties for free: Fetch at a given offset always returns the same record, a restarted broker never renumbers history, and every replica agrees on where a partition's log starts and ends. See Architecture for how the Produce/Fetch dispatch path resolves an offset into a Sequence-bounded read.

Consumer-group, config, and coordinator channels

Beyond the per-partition data logs, the connector maintains four internal channel families — one per coordinator store — that never carry topic data and are never directly reachable by a Kafka or native client. Together with the data channels above, this is the complete channel grammar the connector produces:

{gen} is an internal generation number the snapshot-and-roll mechanism advances as each store's log accumulates writes — it is not something a client ever names or negotiates.

Cross-protocol interoperability

Because a topic's partition-0 log is a normal KubeMQ Events Store channel, a Kafka Produce to topic orders is consumable by a native gRPC/REST Events Store subscriber on channel kafka.orders — and, for a multi-partition topic, on kafka.orders~1, kafka.orders~2, and so on for every additional partition. The reverse direction is symmetric: a native Array.SendEventsStore write to kafka.orders is a legal record any Kafka consumer can Fetch. The four coordinator channel families above are the one asymmetry — they exist so OffsetCommit, group membership, and transaction state stay internal and protocol-correct, not because the underlying store can't hold them like any other channel.

Related