Deployment & High Availability

spec.store.path on the next engine: the operator rejects a leading /. The operator itself supplies the mount-rooted absolute path for the PVC, so a spec.store.path value starting with / is rejected outright — the operator, not the server, owns that absolute path. This is the inverse of the server-layer behavior documented on Storage & Queues: the next engine's server process honors an absolute StorePath verbatim (it's the legacy engine that rewrites a leading / to ./). Keep the two facts distinct — "the operator rejects a leading / in the CR field" and "the next-engine server process honors an absolute StorePath" are both true, at different layers.

/health (liveness) vs /ready (readiness). The table above covers the optional /health liveness probe (spec.health.enabled, off by default). Readiness is separate: for a clustered next-engine deployment (engine=next and not standalone), the operator wires a readinessProbe against a /ready endpoint on the API port — it holds the pod not-Ready until cluster quorum forms, so traffic only reaches pods that can actually serve. Standalone and legacy-engine pods run a single/loopback node with no multi-node quorum to wait on, so the operator does not render this readinessProbe for them. The response code varies by mode, so it isn't listed here.

spec.statefulsetConfigData is a one-way door for every other packaging field. It does not merge with, patch, or extend the StatefulSet the operator builds — it replaces the body outright. The moment it is set, spec.image, spec.volume, spec.resources, spec.health, spec.nodeSelectors, spec.podAntiAffinity and spec.terminationGracePeriodSeconds stop having any effect on the pod, silently: they stay in the CR, they still validate, and they change nothing. You now own the whole pod spec, including the parts the operator was maintaining for you across upgrades. Use it only when a field you need genuinely has no typed equivalent, and expect to re-check it on every operator upgrade.

Only the exposure subset of spec.grpc / spec.rest / spec.api lives here. Their tuning fields (buffer/body limits, gRPC reflection, REST read/write timeouts, CORS, API allow-origins, and the API-auth block) are documented on the Interfaces and Security pages.

spec.replicas becomes immutable once the engine is next — and next is what a new cluster gets. This is a one-way door, armed at creation on a default install, not an edge case for people who opted into next. Once the engine is established as next, a CEL rule on the CRD and an operator fail-closed guard both reject any change to the replica count — up or down. helm upgrade --set replicas=5 fails; so does editing the CR.

Choose the replica count before you create the cluster. 3 is the smallest count that gets a disruption budget; see Disruption budget & pod spread. Changing it afterwards means creating a new cluster and migrating, so decide with the ceiling in mind rather than the starting load.

On Kubernetes, high availability is spec.replicas (operator-managed clustering) — not the config.yaml-only cluster.* block, which has no Helm/CRD path. See the Advanced reference for the Docker cluster.* path.

Use an odd replica count, 3 or more. The operator refuses to protect anything smaller, and tells you. A budget is created only from 3 replicas up — below that the operator creates none and emits an UnsupportedReplicaCount Warning event explaining why: a single replica forms a quorum of one and survives its own restart, while two replicas need both, so any single failure loses quorum. The only correct budget at 2 replicas would be minAvailable: 2, which blocks every voluntary eviction and wedges node drains indefinitely — the operator refuses to ship one that deadlocks maintenance.

An even replica count gets an EvenReplicaCount Warning event: the majority of 4 is 3, the same single-failure tolerance as 3, at the cost of an extra replica and another copy of the data.

These two events are your only signal — nothing else reports that no budget was created:

kubectl get events -n <namespace> --field-selector reason=UnsupportedReplicaCount

The anti-affinity trade is real — choose it deliberately. required: false (the default) means pods always schedule, but two replicas may share a node, so a single node loss can still cost quorum. required: true guarantees the spread, but a replica stays Pending while there are fewer schedulable nodes than replicas.

This setting governs message durability on shutdown, not just tidiness. The server splits the grace period between connector teardown — which requeues in-flight messages that would otherwise be lost — and store shutdown.

Kubernetes does not expose terminationGracePeriodSeconds to the container, so the operator both sets the pod spec and passes the same value as KUBEMQ_TERMINATION_GRACE_PERIOD_SECONDS. Left unset, Kubernetes uses 30s and the server assumes 30s.

The concrete numbers: at the default 30-second grace the requeue backstop is 1 second. 45 seconds funds the full 12-second backstop. Set at least 45 on any cluster where losing in-flight messages on a pod roll matters, and higher still if your connectors hold long in-flight batches.

terminationGracePeriodSeconds: 45

Connector Services & ports (operator-exposed). The pod always publishes container ports for every wire connector — MQTT 1883/8883/8083, AMQP 5672/5671, STOMP 61613/61614, AWS 4566, GCP 8085, Kafka 9092/9093 (deployment/statefulset.go:70-102, deployment/service.go:192-256). The matching ClusterIP Service is created only when that connector is enabled (spec.<connector>.enabled: true) — otherwise the operator prunes it (deployment.go:122-199). Wire connectors are opt-in (enabled: true, default off); the HTTP interfaces above (gRPC/REST/API) remain opt-out (disabled: true).

status.engine. The KubemqCluster status subresource exposes status.engine (legacy / next / auto) — the engine established for this cluster by the guard below. Read it alongside status.replicas when auditing the replicas-freeze caveat above: once status.engine reports next, spec.replicas can no longer change.

auto is not an engine. It means the cluster delegated the choice to the server, which resolves it at boot from the store directory. On such a cluster status.engine and the established-engine annotation both read auto, and the engine actually running is reported only in the server's boot NOTICE in the pod log — see Which engine am I actually on?.

The operator records the live persistence engine in the core.k8s.kubemq.io/established-engine annotation (legacy / next / auto) — once present, this annotation is authoritative on an established cluster, outranking spec.store.engine for every guard decision. A mismatch between the two is refused, not silently applied. Derivation runs in order: (1) the annotation, if present; (2) else the explicit spec.store.engine; (3) else the pod ConfigMap's STORE_ENGINE key; (4) else, if a StatefulSet or retained PVC already exists, the engine is unknown and the operator refuses to guess — set spec.store.engine explicitly or delete the retained PVCs; (5) else the cluster has no engine on record and the operator delegates: it writes STORE_ENGINE=auto and the server resolves the engine at boot from the store directory (clean ⇒ next).

From operator v2.3.0 a fresh cluster gets auto, not a pinned engine — the operator no longer inverts the server's clean-store default. An established cluster keeps its named engine and nothing rolls. Setting spec.store.engine later on an auto cluster is allowed, because auto records a delegation rather than an established engine — pin the engine the server actually resolved; a pin that disagrees with the data on disk is refused by the server at boot, and the server never deletes a datadir.

The operator never changes a live cluster's engine. Two advisory CEL rules on the CRD back this: spec.store.engine is immutable once set (born-one-mode), and spec.replicas is immutable once the engine is established as next.

This operator-side derivation runs before the server-side probe documented on Storage Engines and the Kafka callout on Connectors — two layers of one decision: the operator decides what the pod's env will say before the pod ever boots, and when that env says auto, the server-side probe is what actually decides once it does.