# Three-node cluster with Docker Compose (/deploy/docker-cluster)



<Callout type="warn">
  For **evaluating** clustering, failover and Kafka consumer groups on one machine. Production
  clusters run on Kubernetes with [Helm and the operator](/deploy/kubernetes-helm) — three
  containers on one host share its disk and its fate.
</Callout>

A single container is enough to try KubeMQ ([Quickstart](/deploy/quickstart)). A cluster is
what you need to see leader election, replication and a Kafka consumer group surviving the
loss of a node.

## What each node needs [#what-each-node-needs]

**Two membership lists, and both must be right.** They are separate planes:

| Setting                     | Port | Carries                                                                |
| --------------------------- | ---- | ---------------------------------------------------------------------- |
| `CLUSTER_ROUTES`            | 5228 | Every request forwarded between nodes — including consumer-group joins |
| `CLUSTER_REPLICATION_PEERS` | 6800 | The replicated log (the Raft data plane)                               |

A node whose routes resolve only to itself still joins the replication group and still reports
ready, while being unable to forward anything — so a cluster can look healthy and not be one.
From next v1.2.0 the server catches this itself: `/ready` stops answering 200 and its body
names the forwarding mesh as the reason.

**An identity per node:** a unique `HOST` and a unique `CLUSTER_REPLICATION_REPLICA_ID`
(1, 2, 3 — the id in front of each entry in the peers list).

**A volume per node.** Without one, a node that restarts has lost the replicated log it is
supposed to bring back.

**Three more settings for Kafka clients**, which a single node does not need:

* a distinct `CONNECTORS_KAFKA_PORT` per node, published on the same host port — every node
  advertises its own address, so they cannot share one;
* `CONNECTORS_KAFKA_PEERS`, listing all three as `id@host:port` — a client bootstrapped to one
  node learns the others only from this list;
* the addresses in that list must be the ones **clients** dial, which is why they are
  `127.0.0.1` here and not the container names;
* `CONNECTORS_KAFKA_ADVERTISED_HOST` set to the **same host** as this node's entry in that list.
  Without it the node falls back to advertising its `HOST` name, and logs a warning that its peers
  entry and its advertised address disagree.

<Callout type="info" title="Write 127.0.0.1, not localhost">
  The peers list is handed to clients. On a dual-stack host a client resolving `localhost` tries
  `::1` first, while the published ports are IPv4 only — a stream of "Connection refused" lines
  before each request eventually succeeds on a retry. Use the address clients should actually
  dial: `127.0.0.1` for clients on this host, or the machine's real host name or address for
  clients elsewhere.
</Callout>

## The Compose file [#the-compose-file]

Download [docker-compose-cluster.yml](/docker-compose-cluster.yml), or copy it from here:

```yaml title="docker-compose-cluster.yml"
services:

  kubemq-next-0:
    image: europe-docker.pkg.dev/kubemq/images/kubemq-next:latest
    container_name: kubemq-next-0
    hostname: kubemq-next-0
    environment:
      - KUBEMQ_LICENSE_KEY=${KUBEMQ_LICENSE_KEY:?set KUBEMQ_LICENSE_KEY in your shell}
      - HOST=kubemq-next-0
      - CLUSTER_ENABLE=true
      - CLUSTER_NAME=kubemq-next
      - CLUSTER_PORT=5228
      - CLUSTER_ROUTES=kubemq-next-0:5228,kubemq-next-1:5228,kubemq-next-2:5228
      - CLUSTER_REPLICATION_REPLICA_ID=1
      - CLUSTER_REPLICATION_PEERS=1@kubemq-next-0:6800,2@kubemq-next-1:6800,3@kubemq-next-2:6800
      - CONNECTORS_KAFKA_PORT=19092
      - CONNECTORS_KAFKA_ADVERTISED_HOST=127.0.0.1
      - CONNECTORS_KAFKA_PEERS=1@127.0.0.1:19092,2@127.0.0.1:19093,3@127.0.0.1:19094
    volumes:
      - store_next_0:/kubemq/store
    ports:
      - "50000:50000"   # gRPC
      - "8080:8080"     # API & dashboard
      - "19092:19092"   # Kafka
    networks: [net_next]

  # kubemq-next-1 and kubemq-next-2 repeat this block with REPLICA_ID 2 and 3,
  # Kafka ports 19093 and 19094, and host ports 50001/8081 and 50002/8082.

volumes:
  store_next_0:
  store_next_1:
  store_next_2:

networks:
  net_next:
    driver: bridge
```

## Start it [#start-it]

```bash title="Terminal"
export KUBEMQ_LICENSE_KEY=YOUR_LICENSE_KEY
docker compose -f docker-compose-cluster.yml up -d
```

<Callout type="warn">
  **Start all three together.** A node blocks until two of three have agreed on a leader, and
  gives up after `CLUSTER_REPLICATION_BOOT_TIMEOUT_SECONDS` (60 by default; raise it on a slow
  machine). Starting them one at a time spends that budget on the first node.
</Callout>

## Check that it is one cluster [#check-that-it-is-one-cluster]

```bash title="Terminal"
curl -s localhost:8080/ready    # repeat on 8081 and 8082 — all three must answer ready
```

One of the three reports the leader role and the other two report follower. Then check the
Kafka view, from a client on the host:

```bash title="Terminal"
kcat -b 127.0.0.1:19092 -L                       # must list all three brokers, not one
kcat -b 127.0.0.1:19092 -t demo -P <<< "hello"   # produce through node 0
kcat -b 127.0.0.1:19094 -G demo-group demo -e    # consume in a group through node 2
```

The third command is the one that matters: a consumer group is served by one node on behalf of
the whole cluster, so a group joined through a different node than the one that leads it
exercises both membership lists at once.

## Failover [#failover]

Kill the leader and watch clients continue:

```bash title="Terminal"
docker kill kubemq-next-0
curl -s localhost:8081/ready     # a survivor takes the leader role
```

Clients reconnect on their own; a consumer group rebalances onto the surviving nodes.
Bring the node back with `docker start kubemq-next-0` — it rejoins and catches up from its
volume.

## When something is wrong [#when-something-is-wrong]

| Symptom                                                       | Cause                                                                                                                                                                     |
| ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Produce and plain consume work, only **consumer groups** hang | The address a client is told to use for the group's coordinator is not one it can reach — check `CONNECTORS_KAFKA_PEERS` against the addresses your client actually dials |
| `kcat -L` lists **one** broker                                | `CONNECTORS_KAFKA_PEERS` is missing or lists only this node                                                                                                               |
| Repeated "Connection refused" before requests succeed         | `localhost` in the peers list resolving to `::1`; use `127.0.0.1`                                                                                                         |
| A node never becomes ready                                    | The other two were not started with it, or the routes and peers lists disagree between nodes                                                                              |
| `/ready` fails with a mesh reason in its body (next v1.2.0+)  | That node cannot reach the others on port 5228 — routes, or a blocked port                                                                                                |

## Clients in other containers [#clients-in-other-containers]

The peers list above is written for clients on the **host**. A client in another container is
told the same `127.0.0.1` addresses and will reach itself instead of the broker. Either run
that container with host networking, or write both the peers list and the published ports in
terms of an address both sides share — the host's own name or LAN address.

## Related [#related]

* [Install with Docker](/deploy/docker) — single node, configuration, persistence, upgrades
* [Reaching Kafka on Kubernetes](/connectors/kafka/how-to/kubernetes-access) — the same
  address question, on Kubernetes
* [Install with Helm and the operator](/deploy/kubernetes-helm) — the production path