When work piles up faster than one worker can handle it, you have two completely different levers — and reaching for the wrong one quietly breaks your system. This page is about telling them apart: scaling out (adding workers to share the load) and flow control (keeping a fast producer from drowning a slow consumer).

Picture a busy coffee shop. To serve more customers, you put more baristas behind one counter — each drink order goes to whichever barista is free. That is scaling by competing consumers. Now picture the shop's radio: every barista hears the same announcement, no matter how many you hire. That is fan-out. Adding baristas speeds up the counter; it does nothing to the radio. Confusing the two is how you end up processing every order three times — or building a "load balancer" that never balances.

Competing consumers vs fan-out — the idea

Both shapes start with one stream of messages and several consumers. The difference is who gets each message.

Competing consumers (point-to-point): the consumers share a single logical destination, and each message is handed to exactly one of them. Add a consumer and total throughput goes up, because the work is split. This is how you scale processing.

Fan-out (pub/sub): every consumer is an independent subscriber, and each message is copied to all of them. Add a subscriber and you get one more full copy of the stream — useful for independent reactions, not for sharing load.

Competing consumers: one destination, each message to exactly one worker — add workers to share the load.

Fan-out: one channel, every subscriber gets its own copy — add subscribers for independent reactions, not for throughput.

Consumer groups: getting both from one channel

What if you want load-balancing and broadcast on the same channel? A consumer group is the bridge. Each consumer declares a group name when it subscribes. Within a group, members compete — a message goes to exactly one of them. Across groups, each group gets its own copy.

So three workers in group billing split the stream between them, while a separate analytics group (its own members) receives the full stream in parallel. One channel, two behaviors, decided entirely by group membership.

The group mechanics on this page apply to both Events and Events Store, but durability does not: Events groups are fire-and-forget — a member that is offline simply misses messages, with nothing to replay. Events Store groups are durable and position-tracked — the store remembers each group's progress, so a member that reconnects resumes from where it left off instead of losing messages. The example below uses plain Events; see In KubeMQ for the Events Store distinction in full.

Group billing competes for messages (load-balanced); group analytics gets its own full copy — group name decides the behavior.

Backpressure and slow consumers

Scaling out assumes consumers can keep up. When they cannot — a producer bursts, a downstream API slows down, a worker stalls on a long job — the system needs a way to push back. That feedback is backpressure: signalling upstream to slow down (or buffer, or shed load) so a fast producer does not overwhelm a slow consumer.

Without backpressure, the gap has to go somewhere, and every option is bad: an unbounded in-memory buffer grows until the process runs out of memory; a fixed buffer overflows and silently drops messages; or the producer blocks and the whole pipeline stalls. The healthy outcome is the producer feeling resistance and easing off — exactly like water pressure backing up a pipe when the drain is too small.

A buffer absorbs short bursts; when it fills, backpressure flows back to the producer so it eases off instead of overflowing.

A durable queue is itself a form of backpressure-by-buffering: it absorbs bursts on disk so producers never block on slow consumers, and you drain the backlog by adding more competing workers. A broadcast (fire-and-forget) channel has no such buffer — a subscriber that cannot keep up simply misses messages.

Visibility timeout and in-flight messages

There is a subtler flow-control problem hiding inside competing consumers: what happens to a message while a worker is processing it? If the destination handed the same message to a second worker, you would process it twice. If it deleted the message immediately on delivery, a crash mid-processing would lose it.

The standard answer is the visibility timeout. When a worker receives a message, the message is not deleted — it is hidden from other consumers for a bounded window and counts as in-flight. The worker has until the timeout to finish and acknowledge (ack), which deletes it. If the worker crashes or the timeout expires first, the message becomes visible again and is redelivered to another worker.

Visibility timeout: a received message is in-flight and hidden; ack within the window deletes it, otherwise it reappears for another worker.

The timeout is a balance. Too short, and a legitimately slow job gets redelivered (and processed twice) before it finishes. Too long, and a crashed worker's messages sit invisible for ages before anyone retries them. The limit on in-flight messages also caps real concurrency: a destination only lets so many messages be in-flight at once, which is itself a backpressure knob.

Precise definitions

Competing consumers (point-to-point): a distribution model where multiple consumers read from one shared destination and each message is delivered to exactly one consumer. Throughput scales with the number of consumers.
fan-out (pub/sub): a distribution model where each message is copied to every independent subscriber on a channel. Adding subscribers adds parallel copies, not shared load.
Consumer Group: a named set of consumers on one channel that compete as a unit — each message goes to one member of the group, while every group on the channel receives its own copy.
backpressure: flow control that signals a producer to slow down (or buffer, or shed) when consumers cannot keep up, preventing overflow and loss.
Visibility timeout: the bounded window during which a received-but-unacknowledged message is hidden from other consumers and counts as in-flight; on expiry without an ack it is redelivered.
In-flight message: a message that has been delivered to a consumer but not yet acknowledged — held, not deleted, so it can be redelivered if processing fails.

Trade-offs

Goal	Reach for	Why	Watch out for
Process more, faster	Competing consumers / a group	Each message done once; add workers to scale	One slow worker holds its in-flight messages until timeout
React independently in N places	fan-out / separate groups	Every consumer sees every message	Adds load, not throughput — N copies of the work
Absorb bursts without dropping	Durable queue (buffer)	Disk soaks up the spike; drain with more workers	Backlog grows if consumers stay too slow — monitor depth
Don't overwhelm a slow consumer	Backpressure	Producer eases off instead of overflowing	Fire-and-forget channels have no buffer — slow subscribers miss messages
Survive crashes mid-processing	Visibility timeout + ack	Unacked work reappears for another worker	Wrong timeout → double-processing (too short) or stalls (too long)

Pitfall — "scaling" a broadcast. Adding subscribers to a fire-and-forget channel does not share the load: every subscriber still receives every message, so you multiply the work instead of dividing it. To actually scale processing, put the consumers in the same group so they compete for messages.

Pitfall — designing for exactly-once consumers. Visibility-timeout redelivery means a consumer can see the same message more than once (a slow job, a crash, an expired timeout). Make handlers idempotent so a redelivery is harmless rather than betting on never seeing a duplicate.

In KubeMQ

KubeMQ exposes both levers directly:

Fan-out vs competing consumers is one parameter. On Events and Events Store, subscribers that pass the same group name compete (each message to one member); subscribers with no group (or different groups) each get a full copy. Same channel, different group argument.
Queues give you the buffer and the visibility timeout. A queue durably stores messages, so producers never block on slow consumers — you scale by running more receivers, and each received message is hidden for its visibility timeout until you ack it, then redelivered if you don't.

The snippet below is the same load-balanced subscribe from the Events Consumer Groups tutorial: several consumers join one group on order-events, and KubeMQ delivers each event to exactly one of them. Drop the group name and the very same subscribers turn into fan-out.

grouped_worker.go

sub, err := client.SubscribeToEvents(ctx, "order-events", "workers",
    kubemq.WithOnEvent(func(event *kubemq.Event) {
        fmt.Printf("Processing: %s\n", string(event.Body))
    }),
)
if err != nil {
    log.Fatal(err)
}
defer sub.Unsubscribe()
// Members sharing group "workers" compete; pass "" for fan-out instead.

grouped_worker.py

from kubemq.pubsub import Client as PubSubClient
from kubemq.pubsub import EventsSubscription, CancellationToken

client = PubSubClient(address="localhost:50000")
client.subscribe_to_events(
    subscription=EventsSubscription(
        channel="order-events",
        group="workers",  # same group -> competing consumers; omit for fan-out
        on_receive_event_callback=lambda e: print(f"Processing: {e.body.decode()}"),
    ),
    cancel=CancellationToken(),
)

grouped_worker.js

const { KubeMQClient } = require("kubemq-js");

const client = new KubeMQClient({ address: "localhost:50000" });
client.subscribeToEvents({
  channel: "order-events",
  group: "workers", // same group -> competing consumers; omit for fan-out
  onEvent: (msg) =>
    console.log(`Processing: ${Buffer.from(msg.body).toString()}`),
  onError: (err) => console.error(err.message),
});

GroupedWorker.java

PubSubClient client = PubSubClient.builder()
    .address("localhost:50000")
    .clientId("worker-1")
    .build();

client.subscribeToEvents(EventsSubscription.builder()
    .channel("order-events")
    .group("workers") // same group -> competing consumers; omit for fan-out
    .onReceiveEventCallback(event ->
        System.out.printf("Processing: %s%n", new String(event.getBody())))
    .build());

GroupedWorker.cs

await using var client = new KubeMQClient(new KubeMQClientOptions());
await client.ConnectAsync();

await foreach (var msg in client.SubscribeToEventsAsync(
    new EventsSubscription { Channel = "order-events", Group = "workers" }))
{
    // same Group -> competing consumers; leave Group unset for fan-out
    Console.WriteLine($"Processing: {Encoding.UTF8.GetString(msg.Body.Span)}");
}

GroupedWorker.kt

val client = PubSubClient("localhost:50000")

client.subscribeToEvents(
    channel = "order-events",
    group = "workers", // same group -> competing consumers; omit for fan-out
    onEvent = { event -> println("Processing: ${String(event.body)}") },
    onError = { err -> System.err.println(err.message) },
)

grouped_worker.cpp

auto client = kubemq::PubSubClient("localhost:50000");

// same group ("workers") -> competing consumers; pass "" for fan-out
client.subscribeToEvents("order-events", "workers",
    [](const kubemq::Event& event) {
        std::cout << "Processing: " << event.body << std::endl;
    },
    [](const std::string& err) { std::cerr << err << std::endl; }
);

grouped_worker.rs

let group = "workers"; // same group -> competing consumers; "" for fan-out
let sub = client
    .subscribe_to_events(
        "order-events",
        group,
        |event| {
            Box::pin(async move {
                println!("Processing: {}", String::from_utf8_lossy(&event.body));
            })
        },
        None,
    )
    .await?;

grouped_worker.rb

client = KubeMQ::PubSubClient.new(address: "localhost:50000", client_id: "worker-1")
cancel = KubeMQ::CancellationToken.new

# group: -> competing consumers; omit group: for fan-out
sub = KubeMQ::PubSub::EventsSubscription.new(channel: "order-events", group: "workers")
client.subscribe_to_events(sub, cancellation_token: cancel) do |event|
  puts "Processing: #{event.body}"
end

grouped_worker.exs

{:ok, client} = KubeMQ.Client.start_link(address: "localhost:50000", client_id: "worker-1")

# group: -> competing consumers; omit group: for fan-out
{:ok, _sub} =
  KubeMQ.Client.subscribe_to_events(client, "order-events",
    group: "workers",
    on_event: fn event -> IO.puts("Processing: #{event.body}") end
  )

The same group switch applies to Events Store (durable, position-tracked groups) and to Queues, where competing receivers share the buffer and each delivery is governed by a visibility timeout you ack to clear.

Scaling & Flow Control

Competing consumers vs fan-out — the idea

Consumer groups: getting both from one channel

Backpressure and slow consumers

Visibility timeout and in-flight messages

Precise definitions

Trade-offs

In KubeMQ

How KubeMQ does this

Events Consumer Groups

Scale Subscribers

Handle Slow Consumers

Visibility Timeout

Extend Visibility

Retry with Backoff

Events Store Consumer Groups

On this page