Architecture

Overview

The KubeMQ Kafka connector is an embedded, wire-protocol bridge inside kubemq-server that speaks the genuine Apache Kafka binary protocol — the same length-prefixed frames, the same request/response pairs, and the same flexible-version encoding real kafka-clients, librdkafka, and franz-go clients already speak. The connector is opt-in (disabled by default) — enable it with CONNECTORS_KAFKA_ENABLE=true (Docker) or spec.kafka.enabled: true (Kubernetes) — and it opens two wire-protocol listeners once turned on. Any unmodified Kafka client connects by repointing bootstrap.servers: no library swap, no code change, no KubeMQ SDK.

Two ideas anchor the whole model:

• Every Kafka topic partition maps onto a native KubeMQ Events Store log — a persistent, ordered, replayable primitive that already exists independent of Kafka.
• A Kafka offset is not a separate number the connector invents and tracks — it tracks that log's own Sequence value directly (the two are one apart — Sequence starts at 1, Kafka offsets at 0).

Because the connector reuses an existing KubeMQ primitive instead of building a parallel storage layer, a produced record is durable, replayable by any other KubeMQ transport, and behaves identically on every node of a cluster the moment it lands.

The wire-protocol listeners

A Kafka client bootstraps against one of two TCP listeners:

• Port 9092 — plaintext. A plain TCP socket that accepts raw, length-prefixed Kafka frames.
• Port 9093 — TLS. The same frame format, wrapped in a TLS listener that reuses the server's global certificate configuration — there is no Kafka-specific certificate option.

Both ports stay closed until the connector is enabled. A disabled connector never binds a socket, so a client dialing 9092 or 9093 against a stock, unconfigured KubeMQ server gets connection-refused, not a half-working listener. See Configuration for the opt-in flag and the security posture the listeners enforce.

Once a connection is open, every request runs the same short pipeline before it ever reaches a handler: the frame is decoded into a length-prefixed Kafka request, its API key and version are checked against the connector's advertised version table (an unsupported version is rejected before any work happens), the request size is checked against a bounded cap, and — for everything except the handful of pre-authentication calls — the caller's authorization is enforced. Only then does the request reach the single dispatch function that routes it by API key.

That dispatch function is deliberately narrow: one function, one switch over the Kafka API key, with more than forty explicit cases. Grouping those cases by what they do gives you the connector's real functional surface:

• Data plane — Produce (key 0) and Fetch (key 1) carry the actual records; ListOffsets (key 2) and OffsetForLeaderEpoch (key 23) answer positional questions about a partition without moving data.
• Group coordination — FindCoordinator (10), JoinGroup (11), SyncGroup (14), Heartbeat (12), and LeaveGroup (13) run the classic consumer-group protocol; OffsetCommit (8) and OffsetFetch (9) persist and read back a group's position. See Consumer Groups for how these five calls fit together.
• Admin — CreateTopics (19), DeleteTopics (20), CreatePartitions (37), DescribeConfigs (32), and DescribeCluster (60) manage topic and cluster metadata.
• Share groups (preview) — ShareGroupHeartbeat (76), ShareFetch (78), and ShareAcknowledge (79) implement the KIP-932 queue-style consumption model, alongside admin/observability counterparts. See Share Groups for the preview scope.

A cluster adds one more responsibility ahead of dispatch: a request that must run on the partition leader — a produce, or a group-coordinator call for a group whose coordinator lives on another node — is transparently forwarded there rather than rejected, so a client that happens to connect to a follower still gets a correct answer.

Topics, partitions, and channels

Once a request clears dispatch, it lands on a channel — the same channel a native KubeMQ Events Store client would use. The mapping is deliberately simple:

• Partition 0 of a topic maps to the Events Store log kafka.{topic}.
• Every partition after 0 gets its own log, named kafka.{topic}~{partition} — the topic name, a literal ~, and the partition number.

A topic starts with one partition (unless CreateTopics asks for more) and can only grow — partitions are added, never removed, up to a hard ceiling of 256. Growing a topic to N partitions simply means N independent, equally-ordered Events Store logs back it, each accepting its own stream of produces.

A Kafka request clears the wire-protocol listener and dispatch, then lands on the Events Store log for its partition — kafka.{topic} for partition 0, kafka.{topic}~{partition} for every partition after it — all backed by the message broker.

The offset a client sees for a record isn't a separate number the connector invents and tracks alongside the log — it tracks that log's own Sequence value directly (KubeMQ numbers Sequence starting at 1; Kafka numbers offsets starting at 0, one apart). That has three practical consequences: an offset is durable — it survives a broker restart, because the log itself is durable; it is restart-stable — the same record always reports the same offset, because nothing recomputes it; and on a cluster it is identical on every node — because every node applies the same replicated log in the same order. This is also why compaction and time/size retention — covered in Durability & Retention — never renumber a surviving record's offset: the offset was never a separate, movable number to begin with.

This page only needs the shape of the mapping. The exact offset arithmetic, and the internal channels a consumer group's offsets and membership live on, are catalogued in Topic Mapping.

The next storage engine

The Kafka connector runs only on KubeMQ's next storage engine. That is not a configuration step you perform yourself: enabling Kafka on a fresh store auto-selects next, with no manual store.engine setting required. See Storage Engines → Zero-config engine selection for the complete decision tree — what happens on a fresh store, an existing store, and an explicit pin. This page only needs the coupling, not the rules.

The coupling exists because next is what makes the two guarantees above possible. Every Events Store log on next is backed by an owned segment log replicated with Dragonboat raft — the same mechanism that makes a Kafka offset identical across every node — and next is also the engine that implements compaction, which several Kafka behaviors (Compacted Topics, the Kafka Connect and Kafka Streams ecosystems) depend on.

Related