# Architecture (/connectors/kafka/concepts/architecture)



## Overview [#overview]

The KubeMQ **Kafka connector** is an embedded, wire-protocol bridge inside kubemq-server that
speaks the genuine Apache Kafka binary protocol — the same length-prefixed frames, the same
request/response pairs, and the same flexible-version encoding real `kafka-clients`,
`librdkafka`, and `franz-go` clients already speak. The connector is **opt-in (disabled by
default)** — enable it with `CONNECTORS_KAFKA_ENABLE=true` (Docker) or `spec.kafka.enabled: true`
(Kubernetes) — and it opens two wire-protocol listeners once turned on. Any unmodified Kafka
client connects by repointing `bootstrap.servers`: no library swap, no code change, no KubeMQ
SDK.

Two ideas anchor the whole model:

* Every Kafka **topic partition** maps onto a native KubeMQ **Events Store** log — a persistent,
  ordered, replayable primitive that already exists independent of Kafka.
* A Kafka **offset** is not a separate number the connector invents and tracks — it tracks that
  log's own `Sequence` value directly (the two are one apart — `Sequence` starts at 1, Kafka
  offsets at 0).

Because the connector reuses an existing KubeMQ primitive instead of building a parallel storage
layer, a produced record is durable, replayable by any other KubeMQ transport, and behaves
identically on every node of a cluster the moment it lands.

## The wire-protocol listeners [#the-wire-protocol-listeners]

A Kafka client bootstraps against one of two TCP listeners:

* **Port 9092** — plaintext. A plain TCP socket that accepts raw, length-prefixed Kafka frames.
* **Port 9093** — TLS. The same frame format, wrapped in a TLS listener that reuses the server's
  global certificate configuration — there is no Kafka-specific certificate option.

Both ports stay closed until the connector is enabled. A disabled connector never binds a socket,
so a client dialing 9092 or 9093 against a stock, unconfigured KubeMQ server gets
connection-refused, not a half-working listener. See
[Configuration](/connectors/kafka/concepts/configuration) for the opt-in flag and the
security posture the listeners enforce.

Once a connection is open, every request runs the same short pipeline before it ever reaches a
handler: the frame is decoded into a length-prefixed Kafka request, its API key and version are
checked against the connector's advertised version table (an unsupported version is rejected
before any work happens), the request size is checked against a bounded cap, and — for everything
except the handful of pre-authentication calls — the caller's authorization is enforced. Only
then does the request reach the single **dispatch** function that routes it by API key.

That dispatch function is deliberately narrow: one function, one `switch` over the Kafka API key,
with more than forty explicit cases. Grouping those cases by what they do gives you the
connector's real functional surface:

* **Data plane** — `Produce` (key 0) and `Fetch` (key 1) carry the actual records; `ListOffsets`
  (key 2) and `OffsetForLeaderEpoch` (key 23) answer positional questions about a partition
  without moving data.
* **Group coordination** — `FindCoordinator` (10), `JoinGroup` (11), `SyncGroup` (14), `Heartbeat`
  (12), and `LeaveGroup` (13) run the classic consumer-group protocol; `OffsetCommit` (8) and
  `OffsetFetch` (9) persist and read back a group's position. See
  [Consumer Groups](/connectors/kafka/concepts/consumer-groups) for how these five calls fit
  together.
* **Admin** — `CreateTopics` (19), `DeleteTopics` (20), `CreatePartitions` (37),
  `DescribeConfigs` (32), and `DescribeCluster` (60) manage topic and cluster metadata.
* **Share groups (preview)** — `ShareGroupHeartbeat` (76), `ShareFetch` (78), and
  `ShareAcknowledge` (79) implement the KIP-932 queue-style consumption model, alongside
  admin/observability counterparts. See
  [Share Groups](/connectors/kafka/how-to/share-groups) for the preview scope.

A cluster adds one more responsibility ahead of dispatch: a request that must run on the
partition leader — a produce, or a group-coordinator call for a group whose coordinator lives on
another node — is transparently forwarded there rather than rejected, so a client that happens to
connect to a follower still gets a correct answer.

## Topics, partitions, and channels [#topics-partitions-and-channels]

Once a request clears dispatch, it lands on a channel — the same channel a native KubeMQ Events
Store client would use. The mapping is deliberately simple:

* **Partition 0** of a topic maps to the Events Store log `kafka.{topic}`.
* **Every partition after 0** gets its own log, named `kafka.{topic}~{partition}` — the topic
  name, a literal `~`, and the partition number.

A topic starts with one partition (unless `CreateTopics` asks for more) and can only grow —
partitions are added, never removed, up to a hard ceiling of 256. Growing a topic to `N`
partitions simply means `N` independent, equally-ordered Events Store logs back it, each
accepting its own stream of produces.

<Mermaid
  chart="`
graph TB
CLIENT[&#x22;Kafka client<br/>kafka-clients / librdkafka / franz-go&#x22;]
LISTEN[&#x22;Wire-protocol listeners<br/>:9092 plaintext / :9093 TLS&#x22;]
DISPATCH[&#x22;Dispatch<br/>Produce / Fetch / group coordinator / admin&#x22;]
LOG0[(&#x22;Events Store log<br/>kafka.{topic}&#x22;)]
LOGN[(&#x22;Events Store log<br/>kafka.{topic}~{partition}&#x22;)]
BROKER[&#x22;Message Broker&#x22;]

CLIENT --> LISTEN
LISTEN --> DISPATCH
DISPATCH -- &#x22;partition 0&#x22; --> LOG0
DISPATCH -- &#x22;partition ≥ 1&#x22; --> LOGN
LOG0 --> BROKER
LOGN --> BROKER

class CLIENT client
class LISTEN,DISPATCH connector
class LOG0,LOGN store
class BROKER broker
`"
/>

*A Kafka request clears the wire-protocol listener and dispatch, then lands on the Events Store
log for its partition — `kafka.{topic}` for partition 0, `kafka.{topic}~{partition}` for every
partition after it — all backed by the message broker.*

The offset a client sees for a record isn't a separate number the connector invents and tracks
alongside the log — it tracks that log's own `Sequence` value directly (KubeMQ numbers
`Sequence` starting at 1; Kafka numbers offsets starting at 0, one apart). That has three
practical consequences: an offset is **durable** — it survives a broker restart, because the log
itself is durable; it is **restart-stable** — the same record always reports the same offset,
because nothing recomputes it; and on a cluster it is **identical on every node** — because every
node applies the same replicated log in the same order. This is also why compaction and time/size
retention — covered in
[Durability & Retention](/connectors/kafka/concepts/durability-and-retention) — never
renumber a surviving record's offset: the offset was never a separate, movable number to begin
with.

This page only needs the shape of the mapping. The exact offset arithmetic, and the internal
channels a consumer group's offsets and membership live on, are catalogued in
[Topic Mapping](/connectors/kafka/reference/topic-mapping).

## The next storage engine [#the-next-storage-engine]

The Kafka connector runs **only** on KubeMQ's `next` storage engine. That is not a configuration
step you perform yourself: enabling Kafka on a fresh store &#x2A;*auto-selects `next`**, with no
manual `store.engine` setting required. See
[Storage Engines → Zero-config engine selection](/configure/reference/storage-engines#zero-config-engine-selection)
for the complete decision tree — what happens on a fresh store, an existing store, and an
explicit pin. This page only needs the coupling, not the rules.

The coupling exists because `next` is what makes the two guarantees above possible. Every Events
Store log on `next` is backed by an owned segment log replicated with Dragonboat raft — the same
mechanism that makes a Kafka offset identical across every node — and `next` is also the engine
that implements compaction, which several Kafka behaviors
([Compacted Topics](/connectors/kafka/how-to/compacted-topics), the Kafka Connect and Kafka
Streams ecosystems) depend on.

## Related [#related]

<Cards>
  <Card title="Topic Mapping" href="/connectors/kafka/reference/topic-mapping" description="The exhaustive kafka.{topic} / kafka.{topic}~{partition} grammar, the off-by-one offset-to-Sequence arithmetic, and the group and offset channels." />

  <Card title="Durability & Retention" href="/connectors/kafka/concepts/durability-and-retention" description="The acks contract, quorum-fsynced durability on next, retention by time and size, and log compaction." />

  <Card title="Storage Engines" href="/configure/reference/storage-engines#zero-config-engine-selection" description="The legacy vs next engine model and the zero-config selection rules Kafka relies on." />

  <Card title="Consumer Groups" href="/connectors/kafka/concepts/consumer-groups" description="The classic JoinGroup/SyncGroup/Heartbeat protocol, durable offsets, and static membership." />
</Cards>