Consumer Groups

Overview

The Kafka connector implements Kafka's classic consumer-group protocol — the same FindCoordinator → JoinGroup → SyncGroup → Heartbeat → LeaveGroup sequence every mainstream Kafka client library (kafka-clients, librdkafka, franz-go, sarama) already speaks by default. A group of consumers subscribing to the same topic get their partitions divided among them automatically, their progress recorded durably per group, and their membership rebalanced whenever the group's shape changes. None of this needs a KubeMQ-specific concept: group.id is the only setting a client sets, exactly as it would against real Kafka.

The classic group protocol

Five API calls carry the whole membership lifecycle:

• FindCoordinator locates the broker that owns a given group's state — on a standalone deployment, trivially the node itself.
• JoinGroup is how a consumer enters the group: it names the group, the subscribed topics, and the assignment strategies it supports. The coordinator collects every member's JoinGroup and designates one of them the group leader. A first-time dynamic join without a member ID is briefly turned back with MEMBER_ID_REQUIRED (error code 79) and asked to retry with the ID the broker just minted — a two-round handshake that stops a network-retry storm from registering duplicate phantom members. Static membership, below, is the one case that skips this round-trip entirely.
• SyncGroup is where partitions actually get assigned. The elected leader — not the coordinator — computes the assignment locally, using whichever strategy the group agreed on (range, round-robin, sticky, and so on), and sends that full assignment back through its own SyncGroup call. The coordinator then hands each other member its slice of that same assignment when they call SyncGroup. Assignment is therefore leader-authoritative: the broker distributes an assignment it did not compute.
• Heartbeat keeps a member's membership alive between rebalances; the coordinator times a member out — and removes it — if heartbeats stop arriving within the group's session timeout.
• LeaveGroup is the explicit, immediate departure a client sends on a clean shutdown, so the group doesn't have to wait out a session timeout to notice the member is gone.

Two more calls sit alongside this lifecycle rather than inside it: OffsetCommit and OffsetFetch persist and retrieve a group's position, covered on its own below because it survives across resets of the group protocol itself.

Leader-authoritative assignment and generations

Every completed rebalance bumps the group's generation — a monotonically increasing counter that fences stale requests. A member that heartbeats or commits using a generation number the coordinator no longer recognizes is answered with ILLEGAL_GENERATION (error code 22) rather than silently accepted; this is what stops a member that missed a rebalance from acting on an assignment that no longer applies. Because assignment itself is leader-computed (above), the coordinator's own responsibility across a generation is comparatively narrow: collect membership, forward the leader's assignment to everyone else, and track which generation is current.

Durable per-group offsets

A consumer group's committed position — for every topic-partition it consumes — is stored durably, independent of the group's membership lifecycle. Committing an offset, whether automatically on an interval or explicitly after processing a record, survives every consumer in the group restarting, the group's coordinator failing over to another node, and the group itself going empty and refilling later with a different set of members. See Consuming for the manual-vs-automatic commit mechanics.

Rebalancing

A rebalance reruns JoinGroup/SyncGroup across the whole group, and is triggered by anything that changes what "correct assignment" means:

• A member joins or leaves — explicitly, via LeaveGroup, or implicitly, via a session-timeout.
• A subscribed topic's partition count changes. Growing a topic's partitions is an explicit, operator-triggered action on KubeMQ — partitions only ever increase, never shrink or auto-reshard — and doing so deliberately triggers every subscribed group to rebalance onto the new partition count, rather than leaving some partitions unassigned.

A full rebalance briefly pauses processing for the whole group while JoinGroup/SyncGroup rerun — the cost static membership (below) exists to avoid paying on every routine restart.

Static membership (KIP-345)

A consumer that sets group.instance.id gets a persistent identity the coordinator remembers across disconnects, instead of a fresh, disposable member ID every time it joins:

• Identity, not just a request field. The coordinator maps that instance ID to a specific member ID once and reuses the mapping on every subsequent join from the same instance — the client doesn't need to cache and resend a member ID itself.
• Static join skips the extra round-trip. A dynamic (non-static) member without a member ID gets MEMBER_ID_REQUIRED back, as described above, and has to retry with the ID the broker just handed it. A static member's very first JoinGroup — carrying group.instance.id — resolves straight to its known member ID and is admitted immediately, with no forced retry.
• A clean rejoin skips the rebalance entirely. If the group is already stable, its leader hasn't changed, and a static member rejoins with the same subscription it had before — the common case: the process crashed and restarted, or reconnected after a network blip — the coordinator answers at the same generation with no rebalance at all. Every other member in the group is undisturbed. (One exception: if the rejoining instance is the group's current leader, a full rebalance still runs — the fast path is taken only for non-leader members.)
• A displaced or mismatched instance is fenced. If a second connection shows up claiming an instance ID that's already mapped to a different member ID than the one now presenting it — the classic double-start-during-a-restart shape — the coordinator refuses it with FENCED_INSTANCE_ID (error code 82) rather than quietly admitting a second writer under the same identity.

Static membership is supported in full — every one of the four behaviors above holds across JoinGroup, SyncGroup, Heartbeat, OffsetCommit, and LeaveGroup. It's the mechanism that makes a rolling restart of a consumer fleet cheap: each pod restarts with the same group.instance.id it had before, and the group skips a rebalance for the restart itself — unless the restarting pod is the group's current leader, which still triggers one.

Related