Reaching Kafka on Kubernetes

A KubeMQ cluster on Kubernetes serves Kafka on port 9092 of a Service the operator creates, <cluster>-kafka (for the default Helm install, kubemq-next-kafka.kubemq.svc). Clients inside the cluster use it with no configuration. Reaching it from outside — a laptop, another cluster, a CI runner — needs one decision about addresses, and the obvious shortcut does not work.

Why kubectl port-forward does not work

A Kafka client uses its bootstrap address only once. It connects, asks the broker for the cluster's metadata, and is told the address of every broker and which broker leads each partition. Every later connection goes to those addresses, not to the one you gave it.

On Kubernetes those are in-cluster addresses — the pods' own DNS names. A port-forward carries the first connection to the broker; the client then tries to reach kubemq-next-0.kubemq-next.kubemq.svc from your laptop, cannot resolve it, and every produce and consume fails after a successful connect. The symptom is specific: kcat -L through the port-forward works (it only needs the first connection), and producing or consuming through it does not.

The same is true of Apache Kafka on Kubernetes; it is how the protocol works, not a KubeMQ limit.

Option 1 — run the client inside the cluster

The quickest way to test, and the right answer whenever the application itself will run in the cluster. Nothing to configure:

kubectl run kcat -it --rm --restart=Never --image=edenhill/kcat:1.7.1 -- \
  -b kubemq-next-kafka.kubemq.svc:9092 -L

The operator gives every pod its own advertised address from the StatefulSet's pod DNS names, so a client anywhere in the cluster — any namespace — reaches every broker it is told about.

Option 2 — one node, exposed

A single-node cluster (replicas: 1) has one broker, so one Service address is enough. Expose the Kafka Service and tell the broker to advertise the address clients will use:

spec:
  replicas: 1
  standalone: true
  kafka:
    expose: LoadBalancer              # or NodePort
    advertisedHost: kafka.example.com # the load balancer's DNS name, or a node's IP for NodePort
    advertisedPort: 9092              # the NodePort number when expose is NodePort

advertisedHost is not optional here. Left unset, the broker advertises its in-cluster name, and external clients connect and then hang — exactly the port-forward failure above. If you use the TLS listener on 9093, the server certificate must name the same host.

Option 3 — a cluster, exposed

A multi-node cluster needs one client-reachable address per broker, because a client is sent to whichever broker leads each partition. One Service cannot do that — it is a single address that spreads connections across brokers — so the operator refuses kafka.expose on a clustered cluster unless you also give it per-broker addresses.

Provision one address per pod — for example one LoadBalancer Service per pod, selecting it by the label the StatefulSet puts on every pod — and list them in kafka.peers:

# One per pod: repeat with -1 and -2.
apiVersion: v1
kind: Service
metadata:
  name: kubemq-next-kafka-0
  namespace: kubemq
spec:
  type: LoadBalancer
  selector:
    statefulset.kubernetes.io/pod-name: kubemq-next-0
  ports:
    - name: kafka
      port: 9092
      targetPort: 9092

spec:
  replicas: 3
  kafka:
    # id@host:port, one per broker. The id is the pod's ordinal plus one:
    # kubemq-next-0 is 1, kubemq-next-1 is 2, kubemq-next-2 is 3.
    peers: "1@kafka-0.example.com:9092,2@kafka-1.example.com:9092,3@kafka-2.example.com:9092"

When peers is set, each broker advertises its own entry from it — both in the broker list and when a client asks where its consumer group's coordinator is — and advertisedHost is not used. Every entry must be reachable by every client, including clients inside the cluster: they are told the same addresses.

Checking it from the outside

kcat -b kafka-0.example.com:9092 -L   # lists every broker at the address you configured
kcat -b kafka-0.example.com:9092 -t smoke -P <<< "hello"
kcat -b kafka-0.example.com:9092 -t smoke -C -e

If -L succeeds and produce or consume hangs, the brokers are advertising an address your client cannot reach: compare the addresses -L prints with the ones you can actually connect to.