# Deployment & High Availability (/configure/reference/deployment)
These settings cover how KubeMQ is **packaged and run on Kubernetes** — the container
image, persistent storage, resource requests/limits, health probes, node scheduling, and
`Service` exposure — plus the **high-availability** controls (`replicas` and `standalone`).
Unlike the rest of this reference, these fields do **not** exist in the server's
`config.yaml`: they are typed on the `KubemqCluster` **CRD** and consumed by the operator,
which renders the StatefulSet, Services, and PersistentVolumeClaim. The pattern is therefore
**inverted** — the **Helm/CRD path is the real one**, and the Docker column is `—`
(Kubernetes-only) except where the operator translates a CRD field into a pod env var (shown
as `· env VAR`). On Docker single-node, the equivalent concerns are `docker run` flags
(`-p`, `-v`, `--cpus` / `--memory`) covered in the [Kubernetes guide](/configure/kubernetes).
## Kubernetes packaging [#kubernetes-packaging]
### Container image [#container-image]
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
| ------------------ | ------------- | ------------------------------------------------- | ----------------------------------- | ---------------------------------- | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Image | string | `europe-docker.pkg.dev/kubemq/images/kubemq:next` | image reference | — (K8s-only) | `spec.image.image` | Resolution order: `spec.image.image` → operator env `RELATED_IMAGE_KUBEMQ_CLUSTER` → built-in fallback (`config/image.go:8,18-30`). No server env binding. |
| Pull policy | string (enum) | `Always` | `IfNotPresent` / `Always` / `Never` | — | `spec.image.pullPolicy` | CRD pattern `(IfNotPresent\|Always\|Never)`; empty is coerced to `Always` (`config/image.go:14,34-37`). |
| Image pull secrets | string\[] | `[]` | Secret names | — | — (chart value `imagePullSecrets[]`) | **Not** a typed CRD field — supplied via Helm values / a `spec.statefulsetConfigData` override, not the KubemqCluster spec. |
### Storage (volume) [#storage-volume]
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
| ------------- | ------ | ---------------------------- | -------------------------- | ---------------------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Volume size | string | *unset → ephemeral (no PVC)* | k8s quantity (e.g. `50Gi`) | — | `spec.volume.size` | When set, the operator renders a `volumeClaimTemplates` PVC (`ReadWriteOnce`) mounted at `./kubemq/store`. When **unset/empty, the pod has no persistent volume** and the store is ephemeral (`deployment.go:78-81`, `deployment/statefulset.go:108-124`). There is **no built-in `10Gi` default** in code. **Required for durable `next`-engine data** — a `next` cluster with no `spec.volume.size` set raises the `EphemeralNextStore` warning (its durable data would otherwise live on ephemeral container storage). |
| Storage class | string | `""` → cluster default | StorageClass name | — | `spec.volume.storageClass` | Only consulted when `size` is set; empty renders a blank `storageClassName:` → the cluster's default StorageClass (`config/volume.go:8`). |
**`spec.store.path` on the `next` engine: the operator rejects a leading `/`.** The
operator itself supplies the mount-rooted absolute path for the PVC, so a
`spec.store.path` value starting with `/` is rejected outright — the operator, not
the server, owns that absolute path. This is the **inverse** of the server-layer
behavior documented on [Storage & Queues](/configure/reference/storage-queues):
the `next` engine's **server process** honors an absolute `StorePath` verbatim (it's
the `legacy` engine that rewrites a leading `/` to `./`). Keep the two facts distinct —
"the operator rejects a leading `/` in the CR field" and "the next-engine server
process honors an absolute `StorePath`" are both true, at different layers.
### Resources [#resources]
Each field is rendered into the pod's `resources:` block only when non-empty; there are no
defaults (`config/resources.go:32-47`).
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
| ------------------------- | ------ | ------- | ----------------------------------- | ---------------------------------- | ----------------------------------------- | ------------------------------------------- |
| CPU limit | string | unset | k8s CPU quantity (e.g. `2`, `500m`) | — | `spec.resources.limitsCpu` | Pod `resources.limits.cpu`. |
| Memory limit | string | unset | k8s memory quantity (e.g. `2Gi`) | — | `spec.resources.limitsMemory` | Pod `resources.limits.memory`. |
| Ephemeral-storage limit | string | unset | k8s quantity | — | `spec.resources.limitsEphemeralStorage` | Pod `resources.limits.ephemeral-storage`. |
| CPU request | string | unset | k8s CPU quantity | — | `spec.resources.requestsCpu` | Pod `resources.requests.cpu`. |
| Memory request | string | unset | k8s memory quantity | — | `spec.resources.requestsMemory` | Pod `resources.requests.memory`. |
| Ephemeral-storage request | string | unset | k8s quantity | — | `spec.resources.requestsEphemeralStorage` | Pod `resources.requests.ephemeral-storage`. |
### Health probe [#health-probe]
The liveness probe is **off by default** and only injected when `enabled: true`
(`config/health.go:46-60`).
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
| ----------------- | ----- | ------- | ---------------- | -------------------------------------------------------- | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Enabled | bool | `false` | true / false | · env `API_BIND_ADDRESS=0.0.0.0` (side-effect when true) | `spec.health.enabled` | When true, the operator adds a `livenessProbe` httpGet `/health` on the API port **and** sets `API_BIND_ADDRESS=0.0.0.0` so the kubelet (pod IP) can reach the probe — the API binds `127.0.0.1` otherwise (`config/health.go:48-60`). |
| Initial delay (s) | int32 | `5` | any; `≤0` → `5` | — | `spec.health.initialDelaySeconds` | `config/health.go:38-40`. |
| Period (s) | int32 | `10` | any; `≤0` → `10` | — | `spec.health.periodSeconds` | `config/health.go:35-37`. |
| Timeout (s) | int32 | `5` | any; `≤0` → `5` | — | `spec.health.timeoutSeconds` | `config/health.go:32-34`. |
| Success threshold | int32 | `1` | any; `≤0` → `1` | — | `spec.health.successThreshold` | `config/health.go:29-31`. |
| Failure threshold | int32 | `12` | any; `≤0` → `12` | — | `spec.health.failureThreshold` | `config/health.go:41-43`. |
**`/health` (liveness) vs `/ready` (readiness).** The table above covers the **optional**
`/health` liveness probe (`spec.health.enabled`, off by default). Readiness is separate: for a
**clustered `next`-engine deployment** (`engine=next` and not `standalone`), the operator wires
a **readinessProbe** against a `/ready` endpoint on the API port — it holds the pod not-`Ready`
until cluster quorum forms, so traffic only reaches pods that can actually serve. Standalone and
legacy-engine pods run a single/loopback node with no multi-node quorum to wait on, so the
operator does **not** render this readinessProbe for them. The response code varies by mode, so
it isn't listed here.
### Node scheduling [#node-scheduling]
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
| -------------- | ------------------ | ------- | -------------------------- | ---------------------------------- | ------------------------- | --------------------------------------------------------------------------------------------------------- |
| Node selectors | map\[string]string | `{}` | node label key/value pairs | — | `spec.nodeSelectors.keys` | Rendered as the pod `nodeSelector`; an empty map applies no constraint (`config/node_selectors.go:9-24`). |
### Service exposure & interface toggles [#service-exposure--interface-toggles]
The three built-in interface Services — **gRPC** (`50000`), **REST/WebSocket** (`9090`), and
**API/dashboard** (`8080`) — each expose a Service type, optional NodePort, custom port, and a
disable toggle. These HTTP/interface Services are **opt-out** via `disabled: true`.
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
| ----------------- | ------------- | ------------------- | ----------------------------------------- | ------------------------------------ | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| gRPC disabled | bool | `false` | true / false | · env `CONNECTORS_GRPC_ENABLE=false` | `spec.grpc.disabled` | Deletes the `-grpc` Service and disables the listener (`config/grpc.go:64-66`, `deployment.go:203-206`). |
| gRPC Service type | string (enum) | `ClusterIP` | `ClusterIP` / `NodePort` / `LoadBalancer` | — | `spec.grpc.expose` | Empty → `ClusterIP` (`config/grpc.go:56`); CRD pattern-validated. |
| gRPC NodePort | int32 | `0` (auto-assigned) | 30000–32767 | — | `spec.grpc.nodePort` | Applied only when `expose: NodePort` **and** value > 0 (`config/grpc.go:77-81`). |
| gRPC port | int32 | `50000` | port | · `CONNECTORS_GRPC_PORT` | `spec.grpc.port` | Moves the container/Service/target port and emits the env (`config/grpc.go:71-75`). |
| REST disabled | bool | `false` | true / false | · env `CONNECTORS_REST_ENABLE=false` | `spec.rest.disabled` | The `-rest` Service (9090) is **shared** by REST/MCP/Agents/CE — the operator removes it only when **all four** are disabled (`deployment.go:99-120`). |
| REST Service type | string (enum) | `ClusterIP` | `ClusterIP` / `NodePort` / `LoadBalancer` | — | `spec.rest.expose` | Empty → `ClusterIP` (`config/rest.go:71`). |
| REST NodePort | int32 | `0` (auto) | 30000–32767 | — | `spec.rest.nodePort` | Applied only when `expose: NodePort` and > 0 (`config/rest.go:91-95`). |
| REST port | int32 | `9090` | port | · `CONNECTORS_REST_PORT` | `spec.rest.port` | `config/rest.go:85-88`. |
| API disabled | bool | `false` | true / false | · env `API_ENABLE=false` | `spec.api.disabled` | Deletes the `-api` Service (`config/api.go:65-66`, `deployment.go:89-92`). |
| API Service type | string (enum) | `ClusterIP` | `ClusterIP` / `NodePort` / `LoadBalancer` | — | `spec.api.expose` | Empty → `ClusterIP` (`config/api.go:58`). |
| API NodePort | int32 | `0` (auto) | 30000–32767 | — | `spec.api.nodePort` | Applied only when `expose: NodePort` and > 0 (`config/api.go:79-83`). |
| API port | int32 | `8080` | port | · `API_PORT` | `spec.api.port` | `config/api.go:73-76`. |
**`spec.statefulsetConfigData` is a one-way door for every other packaging field.** It does
not merge with, patch, or extend the StatefulSet the operator builds — it **replaces the
body outright**. The moment it is set, `spec.image`, `spec.volume`, `spec.resources`,
`spec.health`, `spec.nodeSelectors`, `spec.podAntiAffinity` and
`spec.terminationGracePeriodSeconds` stop having any effect on the pod, silently: they stay
in the CR, they still validate, and they change nothing. You now own the whole pod spec,
including the parts the operator was maintaining for you across upgrades. Use it only when
a field you need genuinely has no typed equivalent, and expect to re-check it on every
operator upgrade.
Only the **exposure** subset of `spec.grpc` / `spec.rest` / `spec.api` lives here. Their
tuning fields (buffer/body limits, gRPC reflection, REST read/write timeouts, CORS, API
allow-origins, and the API-auth block) are documented on the
[Interfaces](/configure/reference/interfaces) and
[Security](/configure/reference/security) pages.
### Advanced / other top-level spec fields [#advanced--other-top-level-spec-fields]
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
| -------------------- | ------------------ | ------- | ------------------------- | ---------------------------------- | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| StatefulSet override | string (YAML) | unset | StatefulSet YAML fragment | — | `spec.statefulsetConfigData` | ⚠️ Escape hatch — when set, **replaces the operator-generated StatefulSet body** with your fragment (`deployment/statefulset.go:126-135,268-271`). All other packaging fields above are ignored for the STS. See the warning below. |
| Config data (OIDC) | string | unset | OIDC block | — | `spec.configData` | OIDC-only passthrough — **not** a generic `config.yaml`. See [Security](/configure/reference/security). |
| License | string | unset | license key | — (Secret `LICENSE_KEY_DATA`) | `spec.license` | Injected as Secret data, not env (`deployment.go:66-68`). Config/secret-only. |
| Key | string | unset | activation key | — | `spec.key` | Top-level activation key; secret-only. |
| Key from Secret | string | unset | existing Secret name | — | `spec.keySecretRef` · `spec.keySecretKey` | Sources `key` from an existing Secret instead of the literal. Data key defaults to `key`; override with `keySecretKey`. **Mutually exclusive with the literal `key`** — setting both is rejected. See [Supplying the license from a Secret](/deploy/kubernetes-helm#supplying-the-license-from-a-secret). |
| License from Secret | string | unset | existing Secret name | — | `spec.licenseSecretRef` · `spec.licenseSecretKey` | Same mechanism for `spec.license`; data key defaults to `license`. Mutually exclusive with the literal. |
| Env overlay | map\[string]string | `{}` | env key/value pairs | — | `spec.env` | Last-wins overlay onto the pod ConfigMap — the general escape hatch for any server env key the CRD doesn't type. Operator-computed identity keys are **rejected**: `STORE_ENGINE`, `CLUSTER_ENABLE`, `CLUSTER_NAME`, `CLUSTER_ROUTES`, `API_BIND_ADDRESS`, `CHECKSUM`, `POD_NAME`, and any `CLUSTER_REPLICATION_*` key — **not** a `CLUSTER_*` wildcard (e.g. `CLUSTER_PORT` is still allowed). Not for secrets. |
| Env from Secrets | string\[] | `[]` | existing Secret names | — | `spec.envFromSecrets` | Projects existing Secret(s) as pod env (`envFrom`) — the general Secret-envFrom escape hatch; Secret values never transit the operator. This is the mechanism behind [Kafka SASL credentials](/configure/reference/connectors#kafka-sasl-credentials). |
## High availability [#high-availability]
High availability on Kubernetes is **multiple replicas** managed by the operator — not the
Docker `cluster.*` block.
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
| ------------- | -------- | ------- | --------------------------------------- | ---------------------------------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Replica count | `*int32` | `3` | CRD `Minimum=0`; `0`/nil coerced to `3` | — | `spec.replicas` | nil → 3 (`deployment.go:59-62`), and `SetReplicas(0)` → 3 (`deployment/statefulset.go:183-191`). Backs the `scale` subresource (`specpath=.spec.replicas`). More than one replica = clustered HA (operator wires `CLUSTER_*` + the 5228 cluster port). **Once the engine is established as `next`, `spec.replicas` is immutable** — a CEL rule plus an operator fail-closed guard reject any change (see the engine-establishment guard below). |
| Standalone | bool | `false` | true / false | — | `spec.standalone` | true → a single **non-clustered** node: the operator omits `CLUSTER_NAME`/`CLUSTER_ROUTES`/`CLUSTER_ENABLE` and the `5228` cluster-port from the pod (`deployment/statefulset.go:40-47,103-107`). On the `next` engine, standalone runs a **loopback raft** and skips replication wiring entirely. |
**`spec.replicas` becomes immutable once the engine is `next` — and `next` is what a new
cluster gets.** This is a one-way door, armed at creation on a default install, not an
edge case for people who opted into `next`. Once the engine is established as `next`, a
CEL rule on the CRD and an operator fail-closed guard both reject **any** change to the
replica count — up or down. `helm upgrade --set replicas=5` fails; so does editing the CR.
**Choose the replica count before you create the cluster.** 3 is the smallest count that
gets a disruption budget; see [Disruption budget & pod
spread](#disruption-budget--pod-spread). Changing it afterwards means creating a new
cluster and migrating, so decide with the ceiling in mind rather than the starting load.
On Kubernetes, high availability is `spec.replicas` (operator-managed clustering) — **not**
the `config.yaml`-only `cluster.*` block, which has no Helm/CRD path. See the
[Advanced reference](/configure/reference/advanced) for the Docker `cluster.*` path.
### Disruption budget & pod spread [#disruption-budget--pod-spread]
Both default to **on** for a multi-replica cluster — the operator creates them without
being asked, and these fields exist to tune or disable them.
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
| ------------------------- | ----- | -------------------------------- | ------------ | ---------------------------------- | --------------------------------------- | --------------------------------------------------------------------------------------------- |
| Disruption budget enabled | bool | `true` | true / false | — | `spec.podDisruptionBudget.enabled` | Set `false` to create no `PodDisruptionBudget`. |
| Minimum available | int32 | raft majority (`replicas/2 + 1`) | ≥ `1` | — | `spec.podDisruptionBudget.minAvailable` | Defaults to **2 of 3, 3 of 5**. Overriding it downward trades quorum safety for drainability. |
| Anti-affinity enabled | bool | `true` | true / false | — | `spec.podAntiAffinity.enabled` | Spreads replicas across nodes. |
| Anti-affinity required | bool | `false` | true / false | — | `spec.podAntiAffinity.required` | `false` (default) = *preferred* spread; `true` = *required*. See the trade below. |
**Use an odd replica count, 3 or more. The operator refuses to protect anything smaller,
and tells you.** A budget is created only from **3 replicas up** — below that the operator
creates none and emits an **`UnsupportedReplicaCount`** Warning event explaining why: a
single replica forms a quorum of one and survives its own restart, while two replicas need
both, so any single failure loses quorum. The only correct budget at 2 replicas would be
`minAvailable: 2`, which blocks every voluntary eviction and wedges node drains
indefinitely — the operator refuses to ship one that deadlocks maintenance.
An **even** replica count gets an **`EvenReplicaCount`** Warning event: the majority of 4
is 3, the same single-failure tolerance as 3, at the cost of an extra replica and another
copy of the data.
These two events are your only signal — nothing else reports that no budget was created:
```bash
kubectl get events -n --field-selector reason=UnsupportedReplicaCount
```
**The anti-affinity trade is real — choose it deliberately.**
`required: false` (the default) means pods **always schedule**, but two replicas **may**
share a node, so a single node loss can still cost quorum. `required: true` **guarantees**
the spread, but a replica stays `Pending` while there are fewer schedulable nodes than
replicas.
### Shutdown grace period [#shutdown-grace-period]
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
| ---------------------------- | ----- | -------------------------- | ------------ | ---------------------------------- | ------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------- |
| Termination grace period (s) | int32 | *unset* → Kubernetes' `30` | ≥ `1` | — (`docker stop -t`) | `spec.terminationGracePeriodSeconds` · env `KUBEMQ_TERMINATION_GRACE_PERIOD_SECONDS` | The pod's shutdown budget. The operator sets the **pod spec** and passes the **same number to the server** — they always move together. |
**This setting governs message durability on shutdown, not just tidiness.** The server
splits the grace period between **connector teardown** — which requeues in-flight messages
that would otherwise be **lost** — and **store shutdown**.
Kubernetes does not expose `terminationGracePeriodSeconds` to the container, so the
operator both sets the pod spec **and** passes the same value as
`KUBEMQ_TERMINATION_GRACE_PERIOD_SECONDS`. Left unset, Kubernetes uses **30s** and the
server assumes 30s.
**The concrete numbers:** at the default 30-second grace the requeue backstop is **1
second**. **45 seconds funds the full 12-second backstop.** Set at least `45` on any
cluster where losing in-flight messages on a pod roll matters, and higher still if your
connectors hold long in-flight batches.
```yaml title="values.yaml"
terminationGracePeriodSeconds: 45
```
**Connector Services & ports (operator-exposed).** The pod always publishes container ports
for every wire connector — MQTT `1883/8883/8083`, AMQP `5672/5671`, STOMP `61613/61614`,
AWS `4566`, GCP `8085`, **Kafka `9092/9093`** (`deployment/statefulset.go:70-102`,
`deployment/service.go:192-256`). The matching ClusterIP `Service` is created **only when
that connector is enabled** (`spec..enabled: true`) — otherwise the operator
prunes it (`deployment.go:122-199`). Wire connectors are **opt-in** (`enabled: true`,
default off); the HTTP interfaces above (gRPC/REST/API) remain **opt-out** (`disabled: true`).
**`status.engine`.** The `KubemqCluster` status subresource exposes `status.engine`
(`legacy` / `next` / `auto`) — the engine established for this cluster by the guard below.
Read it alongside `status.replicas` when auditing the replicas-freeze caveat above: once
`status.engine` reports `next`, `spec.replicas` can no longer change.
**`auto` is not an engine.** It means the cluster **delegated** the choice to the server,
which resolves it at boot from the store directory. On such a cluster `status.engine` and
the `established-engine` annotation both read `auto`, and the engine actually running is
reported **only** in the server's boot NOTICE in the pod log — see
[Which engine am I actually on?](/configure/reference/storage-engines#which-engine-am-i-actually-on).
### Engine-establishment guard [#engine-establishment-guard]
The operator records the **live** persistence engine in the
`core.k8s.kubemq.io/established-engine` annotation (`legacy` / `next` / `auto`) — once
present, this annotation is **authoritative** on an established cluster, **outranking
`spec.store.engine`** for every guard decision. A mismatch between the two is **refused,
not silently applied**. Derivation runs in order: (1) the annotation, if present; (2) else
the explicit `spec.store.engine`; (3) else the pod ConfigMap's `STORE_ENGINE` key; (4)
else, if a StatefulSet or retained PVC already exists, the engine is **unknown** and the
operator refuses to guess — set `spec.store.engine` explicitly or delete the retained PVCs;
(5) else the cluster has **no engine on record** and the operator delegates: it writes
`STORE_ENGINE=auto` and the server resolves the engine at boot from the store directory
(clean ⇒ `next`).
**From operator v2.3.0 a fresh cluster gets `auto`, not a pinned engine** — the operator no
longer inverts the server's clean-store default. An established cluster keeps its named
engine and nothing rolls. **Setting `spec.store.engine` later on an `auto` cluster is
allowed**, because `auto` records a delegation rather than an established engine — pin the
engine the server actually resolved; a pin that disagrees with the data on disk is refused
by the server at boot, and the server never deletes a datadir.
The operator **never changes a live cluster's engine.** Two advisory CEL rules on the CRD
back this: `spec.store.engine` is immutable once set (born-one-mode), and `spec.replicas`
is immutable once the engine is established as `next`.
This operator-side derivation runs **before** the server-side probe documented on
[Storage Engines](/configure/reference/storage-engines#zero-config-engine-selection)
and the Kafka callout on [Connectors](/configure/reference/connectors#kafka) — two
layers of one decision: the operator decides what the pod's env will say before the pod
ever boots, and when that env says `auto`, the server-side probe is what actually decides
once it does.
## Example [#example]
Publish the gRPC port on each target. On Kubernetes you set the `Service` exposure and node
port; on Docker you publish the port with `-p`. This is a single-setting snippet — see the
[Kubernetes guide](/configure/kubernetes) for complete, runnable configurations.
```yaml title="values.yaml"
grpc:
expose: NodePort
nodePort: 32000
```
For the full install flow (CRDs → operator → cluster), `values.yaml` mapped to the
`KubemqCluster` spec, and single-node vs HA, see the
[Kubernetes guide](/configure/kubernetes).