KubeMQ
IntegrationsRay ServeScenarios

Kubernetes Production Deployment

Deploy the adapter, a KubeMQ broker, and a Ray cluster to Kubernetes with KEDA autoscaling and secure config.

Scenario

You have a kubemq-rayserve adapter running on your laptop against a local broker, and now it needs to go to production. This scenario takes that exact setup to a Kubernetes cluster: a KubeMQ broker as a StatefulSet, a Ray cluster managed by the KubeRay operator, the adapter deployed as a Ray Serve application, queue-depth autoscaling through KEDA, and credentials kept in a Secret.

Every manifest below is shipped in the integration repo under examples/kubernetes/. They are reference configurations — namespaces, resource limits, storage, and replica counts are deliberately conservative defaults that you tune for your environment.

kubemq-broker.yaml
ray-cluster.yaml
ray-service.yaml
configmap-adapter.yaml
secret-auth.yaml
keda-scaled-object.yaml
helm-values.yaml

Architecture

The producer enqueues tasks onto a KubeMQ queue. Ray Serve workers run the adapter, poll the queue, and process tasks. KEDA watches the same queue's depth through the broker's external scaler and adjusts the worker replica count.

Deploy the KubeMQ Broker

The broker runs as a StatefulSet with persistent storage and a headless Service for peer discovery, plus a stable ClusterIP service (kubemq-cluster-service) that in-cluster clients dial on port 50000. The container exposes three ports: 50000 for gRPC SDK clients, 9090 for the REST gateway and the shared HTTP server that connector endpoints — including the KEDA scaler — use, and 8080 for the management API and dashboard.

kubemq-broker.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kubemq-cluster
  namespace: default
  labels:
    app: kubemq
spec:
  serviceName: kubemq-cluster
  replicas: 1  # Increase for HA; 3 recommended for production
  selector:
    matchLabels:
      app: kubemq
  template:
    metadata:
      labels:
        app: kubemq
    spec:
      containers:
        - name: kubemq
          image: europe-docker.pkg.dev/kubemq/images/kubemq:next
          ports:
            - containerPort: 50000
              name: grpc
            - containerPort: 9090
              name: rest
            - containerPort: 8080
              name: api
          env:
            - name: KUBEMQ_TOKEN
              value: "YOUR-KUBEMQ-LICENSE-KEY"
            - name: CLUSTER_NAME
              value: "kubemq-cluster"
            - name: GRPC_PORT
              value: "50000"
            - name: CONNECTORS_REST_PORT
              value: "9090"
            - name: API_PORT
              value: "8080"
          resources:
            requests:
              cpu: "250m"
              memory: "256Mi"
            limits:
              cpu: "1"
              memory: "1Gi"
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
          volumeMounts:
            - name: kubemq-data
              mountPath: /store
  volumeClaimTemplates:
    - metadata:
        name: kubemq-data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi
terminal
kubectl apply -f examples/kubernetes/kubemq-broker.yaml

Set KUBEMQ_TOKEN to a valid license key before applying. For high availability, raise replicas to 3 and uncomment the CLUSTER_ROUTES env var in the full manifest so the StatefulSet pods discover each other.

Deploy Ray

You can deploy Ray two ways. A plain RayCluster gives you a head node and a worker group that you submit applications to yourself. A RayService goes further: it provisions the cluster, deploys the Serve application graph, monitors health, and performs zero-downtime upgrades. Both require the KubeRay operator. Use a RayCluster when you manage the Serve app separately, and a RayService to let KubeRay own the adapter's full lifecycle.

The cluster's head node runs the Ray GCS, dashboard (8265), client server (10001), and Serve HTTP proxy (8000). Both head and workers receive the broker address through KUBEMQ_ADDRESS, pointing at the ClusterIP service from the broker manifest.

ray-cluster.yaml
apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: kubemq-rayserve-cluster
  namespace: default
  labels:
    app: kubemq-rayserve
spec:
  rayVersion: "2.43.0"
  headGroupSpec:
    rayStartParams:
      dashboard-host: "0.0.0.0"
    template:
      spec:
        containers:
          - name: ray-head
            image: rayproject/ray:2.43.0-py311
            ports:
              - { containerPort: 6379, name: gcs-server }
              - { containerPort: 8265, name: dashboard }
              - { containerPort: 10001, name: client }
              - { containerPort: 8000, name: serve }
            env:
              - name: KUBEMQ_ADDRESS
                value: "kubemq-cluster-service.default.svc.cluster.local:50000"
  workerGroupSpecs:
    - groupName: default-worker
      replicas: 2
      minReplicas: 1
      maxReplicas: 10
      rayStartParams: {}
      template:
        spec:
          containers:
            - name: ray-worker
              image: rayproject/ray:2.43.0-py311
              env:
                - name: KUBEMQ_ADDRESS
                  value: "kubemq-cluster-service.default.svc.cluster.local:50000"
              lifecycle:
                preStop:
                  exec:
                    command: ["/bin/sh", "-c", "ray stop"]

RayService embeds the deployment graph in serveConfigV2 and manages the underlying cluster in rayClusterConfig. The runtime_env installs kubemq-rayserve and the kubemq SDK on workers and injects the broker address, so the adapter is ready the moment the application starts.

ray-service.yaml
apiVersion: ray.io/v1
kind: RayService
metadata:
  name: kubemq-rayserve-service
  namespace: default
  labels:
    app: kubemq-rayserve
spec:
  serviceUnhealthySecondThreshold: 300
  deploymentUnhealthySecondThreshold: 120
  serveConfigV2: |
    applications:
      - name: kubemq-inference
        route_prefix: /
        import_path: your_app.deployment:app
        runtime_env:
          pip:
            - kubemq-rayserve>=1.0.0
            - kubemq>=3.0.0
          env_vars:
            KUBEMQ_ADDRESS: "kubemq-cluster-service.default.svc.cluster.local:50000"
            KUBEMQ_AUTH_TOKEN: ""
        deployments:
          - name: KubeMQTaskConsumer
            num_replicas: 2
            max_ongoing_requests: 100
            autoscaling_config:
              min_replicas: 1
              max_replicas: 10
              target_ongoing_requests: 5
              upscale_delay_s: 10
              downscale_delay_s: 60
            ray_actor_options:
              num_cpus: 0.5
              memory: 536870912  # 512 MiB
  rayClusterConfig:
    rayVersion: "2.43.0"
    headGroupSpec:
      rayStartParams:
        dashboard-host: "0.0.0.0"
      template:
        spec:
          containers:
            - name: ray-head
              image: rayproject/ray:2.43.0-py311
              env:
                - name: KUBEMQ_ADDRESS
                  value: "kubemq-cluster-service.default.svc.cluster.local:50000"
    workerGroupSpecs:
      - groupName: default-worker
        replicas: 2
        minReplicas: 1
        maxReplicas: 10
        rayStartParams: {}
        template:
          spec:
            containers:
              - name: ray-worker
                image: rayproject/ray:2.43.0-py311
                env:
                  - name: KUBEMQ_ADDRESS
                    value: "kubemq-cluster-service.default.svc.cluster.local:50000"
terminal
# RayCluster path
kubectl apply -f examples/kubernetes/ray-cluster.yaml

# or, RayService path (recommended for managed lifecycle)
kubectl apply -f examples/kubernetes/ray-service.yaml

Replace your_app.deployment:app with your own Serve deployment graph that wraps KubeMQTaskProcessorAdapter. Match the Ray and Python versions in the image tag (2.43.0-py311) to the rayVersion and your runtime — a mismatch causes workers to fail at startup.

Configure the Adapter

Non-sensitive adapter settings live in a ConfigMap. Each key maps directly to a KubeMQAdapterConfig field and is consumed by the pod with envFrom. The most important keys are the broker address, the task queue name, and the message size limits.

configmap-adapter.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: kubemq-adapter-config
  namespace: default
  labels:
    app: kubemq-rayserve
data:
  # Connection
  KUBEMQ_ADDRESS: "kubemq-cluster-service.default.svc.cluster.local:50000"
  KUBEMQ_CLIENT_ID: ""

  # Queue / channel
  KUBEMQ_TASK_QUEUE: "rayserve-tasks"
  KUBEMQ_RESULT_CHANNEL_PREFIX: "rayserve-result-"

  # Timeouts (seconds)
  KUBEMQ_RESULT_EXPIRY_SECONDS: "3600"
  KUBEMQ_CONSUMER_POLL_TIMEOUT_SECONDS: "1"
  KUBEMQ_SYNC_INFERENCE_TIMEOUT: "30"

  # Message size limits (bytes)
  KUBEMQ_MAX_SEND_SIZE: "4194304"      # 4 MiB
  KUBEMQ_MAX_RECEIVE_SIZE: "4194304"   # 4 MiB

  # TLS (paths only; cert content comes from the Secret)
  KUBEMQ_TLS: "false"
  KUBEMQ_TLS_CERT_FILE: "/etc/kubemq/tls/client.crt"
  KUBEMQ_TLS_KEY_FILE: "/etc/kubemq/tls/client.key"
  KUBEMQ_TLS_CA_FILE: "/etc/kubemq/tls/ca.crt"
pod-spec.yaml (snippet)
envFrom:
  - configMapRef:
      name: kubemq-adapter-config

KUBEMQ_TASK_QUEUE is a shared contract. The value here (rayserve-tasks) must match the queueName in the KEDA ScaledObject — KEDA scales on the depth of the same queue the adapter polls. Change one and you must change the other, or the scaler watches an empty queue.

Secure Secrets

The auth token and TLS certificates are sensitive and belong in a Secret, never the ConfigMap. A single Secret holds the JWT auth token plus the client cert, client key, and CA cert. It is referenced twice: the adapter pod reads auth-token as an env var and mounts the TLS keys as files, and KEDA's TriggerAuthentication pulls the same auth-token so the scaler authenticates to the broker.

secret-auth.yaml
apiVersion: v1
kind: Secret
metadata:
  name: kubemq-adapter-auth
  namespace: default
  labels:
    app: kubemq-rayserve
type: Opaque
data:
  auth-token: "REPLACE_WITH_BASE64_ENCODED_AUTH_TOKEN"
  tls-client-cert: "REPLACE_WITH_BASE64_ENCODED_CLIENT_CERT"
  tls-client-key: "REPLACE_WITH_BASE64_ENCODED_CLIENT_KEY"
  tls-ca-cert: "REPLACE_WITH_BASE64_ENCODED_CA_CERT"

Encode and apply:

terminal
echo -n "your-jwt-token" | base64
cat client.crt | base64 -w 0
cat client.key | base64 -w 0
cat ca.crt | base64 -w 0

kubectl apply -f examples/kubernetes/secret-auth.yaml

Wire the token and certs into the adapter pod:

pod-spec.yaml (snippet)
env:
  - name: KUBEMQ_AUTH_TOKEN
    valueFrom:
      secretKeyRef:
        name: kubemq-adapter-auth
        key: auth-token
volumes:
  - name: kubemq-tls
    secret:
      secretName: kubemq-adapter-auth
      items:
        - { key: tls-client-cert, path: client.crt }
        - { key: tls-client-key, path: client.key }
        - { key: tls-ca-cert, path: ca.crt }
volumeMounts:
  - name: kubemq-tls
    mountPath: /etc/kubemq/tls
    readOnly: true

Never commit real credentials to version control. The placeholders above are base64-encoded markers only. In production use sealed-secrets, external-secrets, or a vault operator to deliver the actual values.

Autoscale on Queue Depth with KEDA

KEDA polls the standalone KubeMQ external scaler (the kubemq-keda-scaler Helm chart, image kubemq/kubemq-keda-scaler, exposing a gRPC Service on port 9090) and resizes the target deployment based on how many messages are waiting in the queue. The scaler in turn dials the broker on gRPC 50000 to read the live Waiting count. The desired replica count is queueDepth / targetWaiting — with targetWaiting: "5" and 40 waiting tasks, KEDA targets 8 replicas, bounded by minReplicaCount and maxReplicaCount.

keda-scaled-object.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: kubemq-rayserve-scaledobject
  namespace: default
  labels:
    app: kubemq-rayserve
spec:
  scaleTargetRef:
    name: ray-worker-deployment  # Replace with your Ray worker Deployment
  minReplicaCount: 1     # Set to 0 for scale-to-zero
  maxReplicaCount: 20
  pollingInterval: 15
  cooldownPeriod: 60
  advanced:
    restoreToOriginalReplicaCount: false
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0   # Scale up immediately
          policies:
            - type: Pods
              value: 4                     # Add up to 4 pods per 15s
              periodSeconds: 15
        scaleDown:
          stabilizationWindowSeconds: 120  # Wait 2 min before scaling down
          policies:
            - type: Percent
              value: 25                    # Remove at most 25% per minute
              periodSeconds: 60
  triggers:
    - type: external
      metadata:
        scalerAddress: "kubemq-keda-scaler.default.svc.cluster.local:9090"  # External scaler Service
        kubemqAddress: "kubemq-cluster-service.default.svc.cluster.local:50000"  # Broker (gRPC)
        queueName: "rayserve-tasks"   # Must match KUBEMQ_TASK_QUEUE
        targetWaiting: "5"            # Waiting messages per replica
      authenticationRef:
        name: kubemq-trigger-auth
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: kubemq-trigger-auth
  namespace: default
spec:
  secretTargetRef:
    - parameter: authToken
      name: kubemq-adapter-auth     # References secret-auth.yaml
      key: auth-token
terminal
kubectl apply -f examples/kubernetes/keda-scaled-object.yaml

The HPA behavior block keeps scaling stable: scaleUp reacts immediately and adds at most 4 pods every 15 seconds during a burst, while scaleDown waits two minutes and removes no more than 25% of replicas per minute to avoid thrashing. The TriggerAuthentication secretTargetRef pulls the broker token from the same Secret the adapter uses, so credentials live in exactly one place.

KEDA requires the standalone KubeMQ external scaler (the kubemq-keda-scaler chart) to be reachable at scalerAddress on port 9090, the broker reachable at kubemqAddress on gRPC 50000, and the target named in scaleTargetRef to exist before the ScaledObject is applied.

Helm-Based Production Values

For repeatable installs, the repo ships a helm-values.yaml that drives a single chart (or umbrella chart with KubeMQ, KubeRay, and KEDA sub-charts) covering every component above. It centralizes the broker license, Ray version and resources, adapter queue settings, and KEDA toggles.

helm-values.yaml
kubemq:
  enabled: true
  image:
    repository: europe-docker.pkg.dev/kubemq/images/kubemq
    tag: next
  replicas: 1                     # 3 recommended for production HA
  licenseKey: ""                  # REQUIRED
  service:
    type: ClusterIP
    ports: { grpc: 50000, rest: 9090, api: 8080 }
  persistence:
    enabled: true
    size: 5Gi

ray:
  enabled: true
  version: "2.43.0"
  image:
    repository: rayproject/ray
    tag: "2.43.0-py311"
  workers:
    default:
      replicas: 2
      minReplicas: 1
      maxReplicas: 10
  serve:
    importPath: "your_app.deployment:app"
    numReplicas: 2
    maxOngoingRequests: 100
    autoscaling:
      enabled: true
      minReplicas: 1
      maxReplicas: 10
      targetOngoingRequests: 5

adapter:
  address: "kubemq-cluster-service:50000"
  auth:
    enabled: false
    secretName: kubemq-adapter-auth
    authTokenKey: auth-token
  tls:
    enabled: false
    secretName: kubemq-adapter-auth
  taskQueue: "rayserve-tasks"
  resultChannelPrefix: "rayserve-result-"
  maxSendSize: 4194304            # 4 MiB
  maxReceiveSize: 4194304         # 4 MiB
  runtimeEnv:
    pip:
      - "kubemq-rayserve>=1.0.0"
      - "kubemq>=3.0.0"

keda:
  enabled: false                  # Enable KEDA-based scaling
  minReplicaCount: 1
  maxReplicaCount: 20
  trigger:
    scalerAddress: "kubemq-keda-scaler.default.svc.cluster.local:9090"      # External scaler Service (port 9090)
    kubemqAddress: "kubemq-cluster-service.default.svc.cluster.local:50000" # Broker (gRPC 50000)
    queueName: "rayserve-tasks"   # Must match adapter.taskQueue
    targetWaiting: "5"            # Waiting messages per replica
  triggerAuth:
    enabled: false
    secretName: kubemq-adapter-auth
    authTokenKey: auth-token
terminal
helm install kubemq-rayserve ./chart -f examples/kubernetes/helm-values.yaml

Health and Observability

The adapter exposes a health check and a metrics snapshot you should wire into Kubernetes and your dashboards.

health_check() returns True when the broker responds to a ping(), making it a natural backing call for a readiness probe — expose it on an HTTP endpoint in your Serve deployment and point readinessProbe at it so traffic only reaches workers with a live broker connection.

deployment.py (snippet)
from ray import serve
from kubemq_rayserve import KubeMQTaskProcessorAdapter

@serve.deployment
class KubeMQTaskConsumer:
    def __init__(self):
        self.adapter = KubeMQTaskProcessorAdapter()
        # ... self.adapter.initialize(config) ...

    async def check_health(self):
        # Ray Serve calls this periodically; raise to mark unhealthy.
        if not self.adapter.health_check():
            raise RuntimeError("KubeMQ broker unreachable")

    def metrics(self) -> dict:
        return self.adapter.get_metrics_sync()

get_metrics_sync() returns an 8-metric dict. Three are live gauges queried from KubeMQ on each call and are the ones worth charting:

Prop

Type

The dict also carries counters and histograms — tasks_enqueued_total, tasks_completed_total, task_processing_duration_seconds, result_storage_retries_total, and consumer_poll_latency_seconds — tracked in-memory per worker. Scrape queue_depth, in_flight, and dlq_depth into Prometheus to build a saturation dashboard: rising queue_depth with flat in_flight signals you have hit maxReplicaCount, and any dlq_depth growth is an alert condition.

Production Checklist

Before going live, confirm each of the following:

  • Auth and TLS are on. Set a real auth_token and enable TLS — flip KUBEMQ_TLS to "true" and mount the certs from the Secret. The defaults ship disabled for local development only.
  • Message sizes fit your payloads. KUBEMQ_MAX_SEND_SIZE and KUBEMQ_MAX_RECEIVE_SIZE default to 4 MiB. Raise both if your inference inputs or results are larger, or tasks will be rejected.
  • Retries and a DLQ are configured. Set max_retries and a failed_task_queue_name so a repeatedly failing task is routed to the dead-letter queue after its retries are exhausted, then monitor dlq_depth.
  • Pick one autoscaler, not both. Use either the KEDA ScaledObject or the in-process kubemq_queue_depth_policy — running both lets two controllers fight over the same replica count.
  • Confirm scale-to-zero expectations. KEDA can scale to zero with minReplicaCount: 0. The in-process policy cannot — kubemq_queue_depth_policy enforces min_replicas=1 and relies on Ray Serve's downscale_delay_s for cooldown. If you need true scale-to-zero, you must use KEDA.

Next Steps

Queue-Depth Autoscaling

How KEDA scales a KubeMQ queue consumer on waiting-message depth, end to end.

Connection & Security

Configure the adapter with an auth token, TLS, and an in-cluster broker address.

Was this page helpful?

GPU Multi-Model Inference Service

Serve multiple GPU-backed models behind separate KubeMQ queues with batching, progress, and DLQ alerting.

.NET Aspire

Provision KubeMQ and configure IKubeMQClient in .NET Aspire apps with health checks, OpenTelemetry, and keyed DI.

On this page

ScenarioArchitectureDeploy the KubeMQ BrokerDeploy RayConfigure the AdapterSecure SecretsAutoscale on Queue Depth with KEDAHelm-Based Production ValuesHealth and ObservabilityProduction ChecklistNext Steps