Three-node cluster with Docker Compose

A single container is enough to try KubeMQ (Quickstart). A cluster is what you need to see leader election, replication and a Kafka consumer group surviving the loss of a node.

What each node needs

Two membership lists, and both must be right. They are separate planes:

A node whose routes resolve only to itself still joins the replication group and still reports ready, while being unable to forward anything — so a cluster can look healthy and not be one. From next v1.2.0 the server catches this itself: /ready stops answering 200 and its body names the forwarding mesh as the reason.

An identity per node: a unique HOST and a unique CLUSTER_REPLICATION_REPLICA_ID (1, 2, 3 — the id in front of each entry in the peers list).

A volume per node. Without one, a node that restarts has lost the replicated log it is supposed to bring back.

Three more settings for Kafka clients, which a single node does not need:

• a distinct CONNECTORS_KAFKA_PORT per node, published on the same host port — every node advertises its own address, so they cannot share one;
• CONNECTORS_KAFKA_PEERS, listing all three as id@host:port — a client bootstrapped to one node learns the others only from this list;
• the addresses in that list must be the ones clients dial, which is why they are 127.0.0.1 here and not the container names;
• CONNECTORS_KAFKA_ADVERTISED_HOST set to the same host as this node's entry in that list. Without it the node falls back to advertising its HOST name, and logs a warning that its peers entry and its advertised address disagree.

The Compose file

Download docker-compose-cluster.yml, or copy it from here:

services:

  kubemq-next-0:
    image: europe-docker.pkg.dev/kubemq/images/kubemq-next:latest
    container_name: kubemq-next-0
    hostname: kubemq-next-0
    environment:
      - KUBEMQ_LICENSE_KEY=${KUBEMQ_LICENSE_KEY:?set KUBEMQ_LICENSE_KEY in your shell}
      - HOST=kubemq-next-0
      - CLUSTER_ENABLE=true
      - CLUSTER_NAME=kubemq-next
      - CLUSTER_PORT=5228
      - CLUSTER_ROUTES=kubemq-next-0:5228,kubemq-next-1:5228,kubemq-next-2:5228
      - CLUSTER_REPLICATION_REPLICA_ID=1
      - CLUSTER_REPLICATION_PEERS=1@kubemq-next-0:6800,2@kubemq-next-1:6800,3@kubemq-next-2:6800
      - CONNECTORS_KAFKA_PORT=19092
      - CONNECTORS_KAFKA_ADVERTISED_HOST=127.0.0.1
      - CONNECTORS_KAFKA_PEERS=1@127.0.0.1:19092,2@127.0.0.1:19093,3@127.0.0.1:19094
    volumes:
      - store_next_0:/kubemq/store
    ports:
      - "50000:50000"   # gRPC
      - "8080:8080"     # API & dashboard
      - "19092:19092"   # Kafka
    networks: [net_next]

  # kubemq-next-1 and kubemq-next-2 repeat this block with REPLICA_ID 2 and 3,
  # Kafka ports 19093 and 19094, and host ports 50001/8081 and 50002/8082.

volumes:
  store_next_0:
  store_next_1:
  store_next_2:

networks:
  net_next:
    driver: bridge

Start it

export KUBEMQ_LICENSE_KEY=YOUR_LICENSE_KEY
docker compose -f docker-compose-cluster.yml up -d

Check that it is one cluster

curl -s localhost:8080/ready    # repeat on 8081 and 8082 — all three must answer ready

One of the three reports the leader role and the other two report follower. Then check the Kafka view, from a client on the host:

kcat -b 127.0.0.1:19092 -L                       # must list all three brokers, not one
kcat -b 127.0.0.1:19092 -t demo -P <<< "hello"   # produce through node 0
kcat -b 127.0.0.1:19094 -G demo-group demo -e    # consume in a group through node 2

The third command is the one that matters: a consumer group is served by one node on behalf of the whole cluster, so a group joined through a different node than the one that leads it exercises both membership lists at once.

Failover

Kill the leader and watch clients continue:

docker kill kubemq-next-0
curl -s localhost:8081/ready     # a survivor takes the leader role

Clients reconnect on their own; a consumer group rebalances onto the surviving nodes. Bring the node back with docker start kubemq-next-0 — it rejoins and catches up from its volume.

When something is wrong

Clients in other containers

The peers list above is written for clients on the host. A client in another container is told the same 127.0.0.1 addresses and will reach itself instead of the broker. Either run that container with host networking, or write both the peers list and the published ports in terms of an address both sides share — the host's own name or LAN address.

Related

• Install with Docker — single node, configuration, persistence, upgrades
• Reaching Kafka on Kubernetes — the same address question, on Kubernetes
• Install with Helm and the operator — the production path