# Kubernetes Production Deployment (/integrations/rayserve/scenarios/kubernetes-production-deployment)
## Scenario [#scenario]
You have a `kubemq-rayserve` adapter running on your laptop against a local broker, and now it needs to go to production. This scenario takes that exact setup to a Kubernetes cluster: a KubeMQ broker as a `StatefulSet`, a Ray cluster managed by the KubeRay operator, the adapter deployed as a Ray Serve application, queue-depth autoscaling through KEDA, and credentials kept in a `Secret`.
Every manifest below is shipped in the integration repo under `examples/kubernetes/`. They are reference configurations — namespaces, resource limits, storage, and replica counts are deliberately conservative defaults that you tune for your environment.
## Architecture [#architecture]
The producer enqueues tasks onto a KubeMQ queue. Ray Serve workers run the adapter, poll the queue, and process tasks. KEDA watches the same queue's depth through the broker's external scaler and adjusts the worker replica count.
## Deploy the KubeMQ Broker [#deploy-the-kubemq-broker]
The broker runs as a `StatefulSet` with persistent storage and a headless `Service` for peer discovery, plus a stable `ClusterIP` service (`kubemq-cluster-service`) that in-cluster clients dial on port `50000`. The container exposes three ports: `50000` for gRPC SDK clients, `9090` for the REST gateway and the shared HTTP server that connector endpoints — including the KEDA scaler — use, and `8080` for the management API and dashboard.
```yaml title="kubemq-broker.yaml"
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: kubemq-cluster
namespace: default
labels:
app: kubemq
spec:
serviceName: kubemq-cluster
replicas: 1 # Increase for HA; 3 recommended for production
selector:
matchLabels:
app: kubemq
template:
metadata:
labels:
app: kubemq
spec:
containers:
- name: kubemq
image: europe-docker.pkg.dev/kubemq/images/kubemq:next
ports:
- containerPort: 50000
name: grpc
- containerPort: 9090
name: rest
- containerPort: 8080
name: api
env:
- name: KUBEMQ_TOKEN
value: "YOUR-KUBEMQ-LICENSE-KEY"
- name: CLUSTER_NAME
value: "kubemq-cluster"
- name: GRPC_PORT
value: "50000"
- name: CONNECTORS_REST_PORT
value: "9090"
- name: API_PORT
value: "8080"
resources:
requests:
cpu: "250m"
memory: "256Mi"
limits:
cpu: "1"
memory: "1Gi"
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
volumeMounts:
- name: kubemq-data
mountPath: /store
volumeClaimTemplates:
- metadata:
name: kubemq-data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 1Gi
```
```bash title="terminal"
kubectl apply -f examples/kubernetes/kubemq-broker.yaml
```
Set `KUBEMQ_TOKEN` to a valid license key before applying. For high availability, raise `replicas` to `3` and uncomment the `CLUSTER_ROUTES` env var in the full manifest so the StatefulSet pods discover each other.
## Deploy Ray [#deploy-ray]
You can deploy Ray two ways. A plain `RayCluster` gives you a head node and a worker group that you submit applications to yourself. A `RayService` goes further: it provisions the cluster, deploys the Serve application graph, monitors health, and performs zero-downtime upgrades. Both require the KubeRay operator. Use a `RayCluster` when you manage the Serve app separately, and a `RayService` to let KubeRay own the adapter's full lifecycle.
The cluster's head node runs the Ray GCS, dashboard (`8265`), client server (`10001`), and Serve HTTP proxy (`8000`). Both head and workers receive the broker address through `KUBEMQ_ADDRESS`, pointing at the `ClusterIP` service from the broker manifest.
```yaml title="ray-cluster.yaml"
apiVersion: ray.io/v1
kind: RayCluster
metadata:
name: kubemq-rayserve-cluster
namespace: default
labels:
app: kubemq-rayserve
spec:
rayVersion: "2.43.0"
headGroupSpec:
rayStartParams:
dashboard-host: "0.0.0.0"
template:
spec:
containers:
- name: ray-head
image: rayproject/ray:2.43.0-py311
ports:
- { containerPort: 6379, name: gcs-server }
- { containerPort: 8265, name: dashboard }
- { containerPort: 10001, name: client }
- { containerPort: 8000, name: serve }
env:
- name: KUBEMQ_ADDRESS
value: "kubemq-cluster-service.default.svc.cluster.local:50000"
workerGroupSpecs:
- groupName: default-worker
replicas: 2
minReplicas: 1
maxReplicas: 10
rayStartParams: {}
template:
spec:
containers:
- name: ray-worker
image: rayproject/ray:2.43.0-py311
env:
- name: KUBEMQ_ADDRESS
value: "kubemq-cluster-service.default.svc.cluster.local:50000"
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "ray stop"]
```
`RayService` embeds the deployment graph in `serveConfigV2` and manages the underlying cluster in `rayClusterConfig`. The `runtime_env` installs `kubemq-rayserve` and the `kubemq` SDK on workers and injects the broker address, so the adapter is ready the moment the application starts.
```yaml title="ray-service.yaml"
apiVersion: ray.io/v1
kind: RayService
metadata:
name: kubemq-rayserve-service
namespace: default
labels:
app: kubemq-rayserve
spec:
serviceUnhealthySecondThreshold: 300
deploymentUnhealthySecondThreshold: 120
serveConfigV2: |
applications:
- name: kubemq-inference
route_prefix: /
import_path: your_app.deployment:app
runtime_env:
pip:
- kubemq-rayserve>=1.0.0
- kubemq>=3.0.0
env_vars:
KUBEMQ_ADDRESS: "kubemq-cluster-service.default.svc.cluster.local:50000"
KUBEMQ_AUTH_TOKEN: ""
deployments:
- name: KubeMQTaskConsumer
num_replicas: 2
max_ongoing_requests: 100
autoscaling_config:
min_replicas: 1
max_replicas: 10
target_ongoing_requests: 5
upscale_delay_s: 10
downscale_delay_s: 60
ray_actor_options:
num_cpus: 0.5
memory: 536870912 # 512 MiB
rayClusterConfig:
rayVersion: "2.43.0"
headGroupSpec:
rayStartParams:
dashboard-host: "0.0.0.0"
template:
spec:
containers:
- name: ray-head
image: rayproject/ray:2.43.0-py311
env:
- name: KUBEMQ_ADDRESS
value: "kubemq-cluster-service.default.svc.cluster.local:50000"
workerGroupSpecs:
- groupName: default-worker
replicas: 2
minReplicas: 1
maxReplicas: 10
rayStartParams: {}
template:
spec:
containers:
- name: ray-worker
image: rayproject/ray:2.43.0-py311
env:
- name: KUBEMQ_ADDRESS
value: "kubemq-cluster-service.default.svc.cluster.local:50000"
```
```bash title="terminal"
# RayCluster path
kubectl apply -f examples/kubernetes/ray-cluster.yaml
# or, RayService path (recommended for managed lifecycle)
kubectl apply -f examples/kubernetes/ray-service.yaml
```
Replace `your_app.deployment:app` with your own Serve deployment graph that wraps `KubeMQTaskProcessorAdapter`. Match the Ray and Python versions in the image tag (`2.43.0-py311`) to the `rayVersion` and your runtime — a mismatch causes workers to fail at startup.
## Configure the Adapter [#configure-the-adapter]
Non-sensitive adapter settings live in a `ConfigMap`. Each key maps directly to a `KubeMQAdapterConfig` field and is consumed by the pod with `envFrom`. The most important keys are the broker address, the task queue name, and the message size limits.
```yaml title="configmap-adapter.yaml"
apiVersion: v1
kind: ConfigMap
metadata:
name: kubemq-adapter-config
namespace: default
labels:
app: kubemq-rayserve
data:
# Connection
KUBEMQ_ADDRESS: "kubemq-cluster-service.default.svc.cluster.local:50000"
KUBEMQ_CLIENT_ID: ""
# Queue / channel
KUBEMQ_TASK_QUEUE: "rayserve-tasks"
KUBEMQ_RESULT_CHANNEL_PREFIX: "rayserve-result-"
# Timeouts (seconds)
KUBEMQ_RESULT_EXPIRY_SECONDS: "3600"
KUBEMQ_CONSUMER_POLL_TIMEOUT_SECONDS: "1"
KUBEMQ_SYNC_INFERENCE_TIMEOUT: "30"
# Message size limits (bytes)
KUBEMQ_MAX_SEND_SIZE: "4194304" # 4 MiB
KUBEMQ_MAX_RECEIVE_SIZE: "4194304" # 4 MiB
# TLS (paths only; cert content comes from the Secret)
KUBEMQ_TLS: "false"
KUBEMQ_TLS_CERT_FILE: "/etc/kubemq/tls/client.crt"
KUBEMQ_TLS_KEY_FILE: "/etc/kubemq/tls/client.key"
KUBEMQ_TLS_CA_FILE: "/etc/kubemq/tls/ca.crt"
```
```yaml title="pod-spec.yaml (snippet)"
envFrom:
- configMapRef:
name: kubemq-adapter-config
```
`KUBEMQ_TASK_QUEUE` is a shared contract. The value here (`rayserve-tasks`) must match the `queueName` in the KEDA `ScaledObject` — KEDA scales on the depth of the *same* queue the adapter polls. Change one and you must change the other, or the scaler watches an empty queue.
## Secure Secrets [#secure-secrets]
The auth token and TLS certificates are sensitive and belong in a `Secret`, never the `ConfigMap`. A single `Secret` holds the JWT auth token plus the client cert, client key, and CA cert. It is referenced twice: the adapter pod reads `auth-token` as an env var and mounts the TLS keys as files, and KEDA's `TriggerAuthentication` pulls the same `auth-token` so the scaler authenticates to the broker.
```yaml title="secret-auth.yaml"
apiVersion: v1
kind: Secret
metadata:
name: kubemq-adapter-auth
namespace: default
labels:
app: kubemq-rayserve
type: Opaque
data:
auth-token: "REPLACE_WITH_BASE64_ENCODED_AUTH_TOKEN"
tls-client-cert: "REPLACE_WITH_BASE64_ENCODED_CLIENT_CERT"
tls-client-key: "REPLACE_WITH_BASE64_ENCODED_CLIENT_KEY"
tls-ca-cert: "REPLACE_WITH_BASE64_ENCODED_CA_CERT"
```
Encode and apply:
```bash title="terminal"
echo -n "your-jwt-token" | base64
cat client.crt | base64 -w 0
cat client.key | base64 -w 0
cat ca.crt | base64 -w 0
kubectl apply -f examples/kubernetes/secret-auth.yaml
```
Wire the token and certs into the adapter pod:
```yaml title="pod-spec.yaml (snippet)"
env:
- name: KUBEMQ_AUTH_TOKEN
valueFrom:
secretKeyRef:
name: kubemq-adapter-auth
key: auth-token
volumes:
- name: kubemq-tls
secret:
secretName: kubemq-adapter-auth
items:
- { key: tls-client-cert, path: client.crt }
- { key: tls-client-key, path: client.key }
- { key: tls-ca-cert, path: ca.crt }
volumeMounts:
- name: kubemq-tls
mountPath: /etc/kubemq/tls
readOnly: true
```
Never commit real credentials to version control. The placeholders above are base64-encoded markers only. In production use sealed-secrets, external-secrets, or a vault operator to deliver the actual values.
## Autoscale on Queue Depth with KEDA [#autoscale-on-queue-depth-with-keda]
KEDA polls the standalone KubeMQ external scaler (the `kubemq-keda-scaler` Helm chart, image `kubemq/kubemq-keda-scaler`, exposing a gRPC Service on port `9090`) and resizes the target deployment based on how many messages are waiting in the queue. The scaler in turn dials the broker on gRPC `50000` to read the live `Waiting` count. The desired replica count is `queueDepth / targetWaiting` — with `targetWaiting: "5"` and 40 waiting tasks, KEDA targets 8 replicas, bounded by `minReplicaCount` and `maxReplicaCount`.
```yaml title="keda-scaled-object.yaml"
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: kubemq-rayserve-scaledobject
namespace: default
labels:
app: kubemq-rayserve
spec:
scaleTargetRef:
name: ray-worker-deployment # Replace with your Ray worker Deployment
minReplicaCount: 1 # Set to 0 for scale-to-zero
maxReplicaCount: 20
pollingInterval: 15
cooldownPeriod: 60
advanced:
restoreToOriginalReplicaCount: false
horizontalPodAutoscalerConfig:
behavior:
scaleUp:
stabilizationWindowSeconds: 0 # Scale up immediately
policies:
- type: Pods
value: 4 # Add up to 4 pods per 15s
periodSeconds: 15
scaleDown:
stabilizationWindowSeconds: 120 # Wait 2 min before scaling down
policies:
- type: Percent
value: 25 # Remove at most 25% per minute
periodSeconds: 60
triggers:
- type: external
metadata:
scalerAddress: "kubemq-keda-scaler.default.svc.cluster.local:9090" # External scaler Service
kubemqAddress: "kubemq-cluster-service.default.svc.cluster.local:50000" # Broker (gRPC)
queueName: "rayserve-tasks" # Must match KUBEMQ_TASK_QUEUE
targetWaiting: "5" # Waiting messages per replica
authenticationRef:
name: kubemq-trigger-auth
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: kubemq-trigger-auth
namespace: default
spec:
secretTargetRef:
- parameter: authToken
name: kubemq-adapter-auth # References secret-auth.yaml
key: auth-token
```
```bash title="terminal"
kubectl apply -f examples/kubernetes/keda-scaled-object.yaml
```
The HPA `behavior` block keeps scaling stable: `scaleUp` reacts immediately and adds at most 4 pods every 15 seconds during a burst, while `scaleDown` waits two minutes and removes no more than 25% of replicas per minute to avoid thrashing. The `TriggerAuthentication` `secretTargetRef` pulls the broker token from the same `Secret` the adapter uses, so credentials live in exactly one place.
KEDA requires the standalone KubeMQ external scaler (the `kubemq-keda-scaler` chart) to be reachable at `scalerAddress` on port `9090`, the broker reachable at `kubemqAddress` on gRPC `50000`, and the target named in `scaleTargetRef` to exist before the `ScaledObject` is applied.
## Helm-Based Production Values [#helm-based-production-values]
For repeatable installs, the repo ships a `helm-values.yaml` that drives a single chart (or umbrella chart with KubeMQ, KubeRay, and KEDA sub-charts) covering every component above. It centralizes the broker license, Ray version and resources, adapter queue settings, and KEDA toggles.
```yaml title="helm-values.yaml"
kubemq:
enabled: true
image:
repository: europe-docker.pkg.dev/kubemq/images/kubemq
tag: next
replicas: 1 # 3 recommended for production HA
licenseKey: "" # REQUIRED
service:
type: ClusterIP
ports: { grpc: 50000, rest: 9090, api: 8080 }
persistence:
enabled: true
size: 5Gi
ray:
enabled: true
version: "2.43.0"
image:
repository: rayproject/ray
tag: "2.43.0-py311"
workers:
default:
replicas: 2
minReplicas: 1
maxReplicas: 10
serve:
importPath: "your_app.deployment:app"
numReplicas: 2
maxOngoingRequests: 100
autoscaling:
enabled: true
minReplicas: 1
maxReplicas: 10
targetOngoingRequests: 5
adapter:
address: "kubemq-cluster-service:50000"
auth:
enabled: false
secretName: kubemq-adapter-auth
authTokenKey: auth-token
tls:
enabled: false
secretName: kubemq-adapter-auth
taskQueue: "rayserve-tasks"
resultChannelPrefix: "rayserve-result-"
maxSendSize: 4194304 # 4 MiB
maxReceiveSize: 4194304 # 4 MiB
runtimeEnv:
pip:
- "kubemq-rayserve>=1.0.0"
- "kubemq>=3.0.0"
keda:
enabled: false # Enable KEDA-based scaling
minReplicaCount: 1
maxReplicaCount: 20
trigger:
scalerAddress: "kubemq-keda-scaler.default.svc.cluster.local:9090" # External scaler Service (port 9090)
kubemqAddress: "kubemq-cluster-service.default.svc.cluster.local:50000" # Broker (gRPC 50000)
queueName: "rayserve-tasks" # Must match adapter.taskQueue
targetWaiting: "5" # Waiting messages per replica
triggerAuth:
enabled: false
secretName: kubemq-adapter-auth
authTokenKey: auth-token
```
```bash title="terminal"
helm install kubemq-rayserve ./chart -f examples/kubernetes/helm-values.yaml
```
## Health and Observability [#health-and-observability]
The adapter exposes a health check and a metrics snapshot you should wire into Kubernetes and your dashboards.
`health_check()` returns `True` when the broker responds to a `ping()`, making it a natural backing call for a readiness probe — expose it on an HTTP endpoint in your Serve deployment and point `readinessProbe` at it so traffic only reaches workers with a live broker connection.
```python title="deployment.py (snippet)"
from ray import serve
from kubemq_rayserve import KubeMQTaskProcessorAdapter
@serve.deployment
class KubeMQTaskConsumer:
def __init__(self):
self.adapter = KubeMQTaskProcessorAdapter()
# ... self.adapter.initialize(config) ...
async def check_health(self):
# Ray Serve calls this periodically; raise to mark unhealthy.
if not self.adapter.health_check():
raise RuntimeError("KubeMQ broker unreachable")
def metrics(self) -> dict:
return self.adapter.get_metrics_sync()
```
`get_metrics_sync()` returns an 8-metric dict. Three are live gauges queried from KubeMQ on each call and are the ones worth charting:
The dict also carries counters and histograms — `tasks_enqueued_total`, `tasks_completed_total`, `task_processing_duration_seconds`, `result_storage_retries_total`, and `consumer_poll_latency_seconds` — tracked in-memory per worker. Scrape `queue_depth`, `in_flight`, and `dlq_depth` into Prometheus to build a saturation dashboard: rising `queue_depth` with flat `in_flight` signals you have hit `maxReplicaCount`, and any `dlq_depth` growth is an alert condition.
## Production Checklist [#production-checklist]
Before going live, confirm each of the following:
* **Auth and TLS are on.** Set a real `auth_token` and enable TLS — flip `KUBEMQ_TLS` to `"true"` and mount the certs from the `Secret`. The defaults ship disabled for local development only.
* **Message sizes fit your payloads.** `KUBEMQ_MAX_SEND_SIZE` and `KUBEMQ_MAX_RECEIVE_SIZE` default to 4 MiB. Raise both if your inference inputs or results are larger, or tasks will be rejected.
* **Retries and a DLQ are configured.** Set `max_retries` and a `failed_task_queue_name` so a repeatedly failing task is routed to the dead-letter queue after its retries are exhausted, then monitor `dlq_depth`.
* **Pick one autoscaler, not both.** Use *either* the KEDA `ScaledObject` *or* the in-process `kubemq_queue_depth_policy` — running both lets two controllers fight over the same replica count.
* **Confirm scale-to-zero expectations.** KEDA can scale to zero with `minReplicaCount: 0`. The in-process policy cannot — `kubemq_queue_depth_policy` enforces `min_replicas=1` and relies on Ray Serve's `downscale_delay_s` for cooldown. If you need true scale-to-zero, you must use KEDA.
## Next Steps [#next-steps]