# Topic Mapping (/connectors/kafka/reference/topic-mapping)



This is the master reference for how the embedded KubeMQ Kafka connector maps Kafka topics,
partitions, and offsets onto KubeMQ channels. Every topic is backed by one or more KubeMQ
**Events Store** logs — one per partition — plus a small family of internal, reserved channels
that hold consumer-group offsets, per-topic config, and transaction/producer state. This
deterministic mapping is what lets a native gRPC/REST Events Store subscriber read the exact
same data a Kafka client produced, on the same cluster, with no translation step.

## Topic grammar [#topic-grammar]

A Kafka topic maps to one KubeMQ Events Store log per partition. Partition 0 gets the bare,
back-compat form; every additional partition appends a reserved separator and its index:

```text
kafka.{topic}
└──┬─┘└──┬──┘
   │     └─ the bare topic id (the name a client passes to Produce/Fetch/CreateTopics)
   └─ fixed connector prefix ("kafka."), partition 0 only

kafka.{topic}~{partition}
└──┬─┘└──┬──┘└┬┘└───┬────┘
   │     │    │     └─ the partition index (1, 2, 3, …)
   │     │    └─ the reserved "~" partition separator
   │     └─ the bare topic id
   └─ fixed connector prefix ("kafka."), partition ≥ 1
```

| Kafka topic | Partition | KubeMQ channel    |
| ----------- | --------- | ----------------- |
| `orders`    | 0         | `kafka.orders`    |
| `orders`    | 1         | `kafka.orders~1`  |
| `orders`    | 5         | `kafka.orders~5`  |
| `audit-log` | 0         | `kafka.audit-log` |

A topic starts at 1 partition (`CreateTopics`'s `NumPartitions`, or the implicit single-partition
default on auto-create) and can only ever **grow** — see
[Partitions & Ordering](/connectors/kafka/concepts/partitions-and-ordering) for the
increase-only mechanism and the per-key ordering guarantees that come with it. Each additional
partition synthesizes as its own independent channel, with its own ordered offset space, its own
log-start, and its own retention/compaction state.

<Callout type="warn">
  **`~` is reserved — a topic name may not contain it.** The connector rejects any topic name
  containing the partition separator at admission (`INVALID_TOPIC_EXCEPTION`), so `kafka.{topic}` and
  `kafka.{topic}~{partition}` can never collide: no legal topic name can produce a channel that looks
  like another topic's partitioned form. Kafka's own charset (`[a-zA-Z0-9._-]`) never uses `~`
  anyway — this only matters for hand-built clients, since the connector is the sole admission gate.
</Callout>

## Offsets are Events Store sequence numbers [#offsets-are-events-store-sequence-numbers]

The connector keeps no separate offset index. A partition's Kafka offsets and its channel's
Events Store `Sequence` numbers are the **same counter**, one apart: KubeMQ numbers `Sequence`
starting at 1, Kafka numbers offsets starting at 0, so for every record:

```text
offset = Sequence − 1        (Sequence = offset + 1)
```

That single fact is what makes a robust drop-in possible without KubeMQ maintaining a shadow
index. Because `Sequence` is durable, restart-stable, and identical across every node of a Raft
cluster, a Kafka offset inherits all three properties for free: `Fetch` at a given offset always
returns the same record, a restarted broker never renumbers history, and every replica agrees on
where a partition's log starts and ends. See
[Architecture](/connectors/kafka/concepts/architecture) for how the Produce/Fetch dispatch
path resolves an offset into a `Sequence`-bounded read.

## Consumer-group, config, and coordinator channels [#consumer-group-config-and-coordinator-channels]

Beyond the per-partition data logs, the connector maintains four internal channel families —
one per coordinator store — that never carry topic data and are never directly reachable by a
Kafka or native client. Together with the data channels above, this is the complete channel
grammar the connector produces:

| Channel type                  | Pattern                               | Example                            | Holds                                                                                                         |
| ----------------------------- | ------------------------------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| Topic log, partition 0        | `kafka.{topic}`                       | `kafka.orders`                     | Produced records for partition 0                                                                              |
| Topic log, partition ≥ 1      | `kafka.{topic}~{partition}`           | `kafka.orders~1`                   | Produced records for that partition                                                                           |
| Consumer-group commit log     | `_KAFKA_OFFSETS_.{group}.{gen}`       | `_KAFKA_OFFSETS_.billing-worker.3` | Durable per-group committed offsets (`OffsetCommit`/`OffsetFetch`), snapshotted and rolled across generations |
| Per-topic config store        | `_KAFKA_CONFIG_.{topic}.{gen}`        | `_KAFKA_CONFIG_.orders.2`          | The topic's partition count and per-topic config overlay (`cleanup.policy`, `retention.ms`, …)                |
| PID-block allocator           | `_KAFKA_PIDS_.{gen}`                  | `_KAFKA_PIDS_.5`                   | Producer-ID blocks for the idempotent and transactional producer (keyless — one shared family, not per-topic) |
| Transaction-coordinator state | `_KAFKA_TXN_.{transactionalID}.{gen}` | `_KAFKA_TXN_.checkout-svc.1`       | Per-`transactional.id` coordinator state — producer epoch, open partitions, commit/abort markers              |

`{gen}` is an internal generation number the snapshot-and-roll mechanism advances as each store's
log accumulates writes — it is not something a client ever names or negotiates.

<Callout type="info">
  **The `_KAFKA_` prefix is a reserved, protected namespace.** Every channel above the data-log rows
  lives under `_KAFKA_`, a namespace the broker rejects writes and subscriptions to from any
  external caller — Kafka client or native gRPC/REST — regardless of authorization policy. Only the
  connector's own internal code path may read or write these channels. This is why `OffsetFetch`,
  `DescribeGroups`, and the consumer-group lag metric exist as dedicated Kafka APIs rather than "just
  subscribe to the offsets channel": there is no wire-visible compacted topic to tail, by design.
</Callout>

## Cross-protocol interoperability [#cross-protocol-interoperability]

Because a topic's partition-0 log is a normal KubeMQ Events Store channel, a Kafka `Produce` to
topic `orders` is consumable by a native gRPC/REST Events Store subscriber on channel
`kafka.orders` — and, for a multi-partition topic, on `kafka.orders~1`, `kafka.orders~2`, and so
on for every additional partition. The reverse direction is symmetric: a native
`Array.SendEventsStore` write to `kafka.orders` is a legal record any Kafka consumer can `Fetch`.
The four coordinator channel families above are the one asymmetry — they exist so `OffsetCommit`,
group membership, and transaction state stay internal and protocol-correct, not because the
underlying store can't hold them like any other channel.

<Callout type="info">
  **Deterministic read.** Subscribe to the Events Store log with start policy `startAt = "new"`
  **before** a Kafka producer's first `Produce`, so the produced record is guaranteed in-window for
  the native consumer — the same no-startup-race pattern every Events Store subscriber follows,
  Kafka-sourced or not.
</Callout>

## Related [#related]

<Cards>
  <Card title="Architecture" href="/connectors/kafka/concepts/architecture" description="The 9092/9093 wire-protocol listeners and how Produce/Fetch dispatch onto these channels." />

  <Card title="Partitions & Ordering" href="/connectors/kafka/concepts/partitions-and-ordering" description="The increase-only partition model and the per-key ordering guarantee behind the ~{partition} suffix." />

  <Card title="Capabilities" href="/connectors/kafka/reference/capabilities" description="Every Kafka API the connector implements, including the OffsetCommit/OffsetFetch and group APIs that read these channels." />

  <Card title="Limits & Rules" href="/connectors/kafka/reference/limits-and-rules" description="The 256-partition cap and the other numeric limits this channel mapping enforces." />
</Cards>