Ray Serve
Run queue-based async and sync ML inference on Ray Serve with KubeMQ as the broker, result backend, autoscaler, and progress bus.
kubemq-rayserve is a Python package that plugs KubeMQ into
Ray Serve as a task backend. It provides
KubeMQTaskProcessorAdapter — the first non-Celery adapter for Ray Serve's
TaskProcessorAdapter framework — so a Ray Serve deployment can run queue-based asynchronous
and blocking synchronous ML inference, with KubeMQ acting as the message broker, the result
store, the autoscaling signal, and the progress event bus all at once.
The adapter is a native gRPC SDK client. It connects to a KubeMQ broker on port 50000 and is
always available — there is no connector to enable on the server. You install one package, point
it at a broker, and Ray Serve drives the rest of the lifecycle.
Ray Serve is an integration: a client-side library that embeds a KubeMQ SDK, distinct from the server-side connectors (MCP, A2A, CloudEvents) that run inside the broker.
Why KubeMQ + Ray Serve
- One Kubernetes-native broker — a single binary carries the task queue, the synchronous query channel, the result store, and the progress event stream. No Erlang, Redis, or RabbitMQ to operate alongside Ray.
- Built-in sync inference —
query_task_syncgives blocking request-response inference that the reference Celery adapter does not offer. - Built-in progress tracking —
report_progressstreams real-time task progress over KubeMQ Events; Celery requires custom signals. - Queue-peek result backend — results live in KubeMQ itself, with no separate Redis or database to provision.
- Native autoscaling —
kubemq_queue_depth_policyscales Ray Serve replicas from live queue depth, and the same depth drives KEDA for cluster-level scaling. - One-dependency setup — only KubeMQ is required, versus a broker plus a separate result backend.
Install
uv pip install kubemq-rayserve| Requirement | Version |
|---|---|
| Python | >= 3.10 |
Ray Serve (ray[serve]) | >= 2.50.0 |
kubemq (Python SDK) | >= 4.1.5 |
pydantic | >= 2.0 |
A running KubeMQ broker is required (default address localhost:50000). Start one locally with
Docker:
docker run -d \ --name kubemq \ -p 50000:50000 \ -p 9090:9090 \ -e KUBEMQ_TOKEN=YOUR_LICENSE_KEY \ europe-docker.pkg.dev/kubemq/images/kubemq:nextPort 50000 is KubeMQ's gRPC endpoint — the only port the adapter's SDK clients use. Port 9090
is the shared HTTP server (REST and the AI-agent connectors); the dashboard runs separately on
port 8080. The adapter and the kubemq_queue_depth_policy autoscaler both speak gRPC on 50000.
Architecture
A producer enqueues a serialized task onto a KubeMQ Queue. Inside the Ray Serve deployment, the
KubeMQTaskProcessorAdapter runs a consumer thread that polls the queue, dispatches each task to a
registered handler, and writes the result back into the queue-peek result backend. A parallel Query
channel serves blocking sync inference, and an Events channel carries progress updates — all over
the same gRPC connection to port 50000.
The adapter maps Ray Serve's task model onto three KubeMQ primitives — Queues, Queries, and Events — over one gRPC connection.
The adapter never re-teaches KubeMQ messaging: async tasks ride Queues, sync inference rides request-response RPC (Queries), and progress rides Events. Each capability page documents the adapter API and links to the underlying concept.
Capabilities
KubeMQTaskProcessorAdapter exposes eight capabilities, grouped into three inference models plus
the operational features that surround them.
| Capability | What it does | KubeMQ primitive |
|---|---|---|
| Async inference | Enqueue a task, poll the result backend for the outcome | Queues |
| Sync inference | Blocking request-response inference via query_task_sync | RPC (Queries) |
| Progress tracking | Stream live report_progress updates to subscribers | Events |
| Cancellation | Soft-cancel a task — overwrite its result with CANCELLED | Queues |
| DLQ monitoring | on_dlq callback alerts on permanently failed tasks | Queues |
| Autoscaling | Scale replicas on queue depth (kubemq_queue_depth_policy / KEDA) | Queues |
| Result backend | Queue-peek result storage with purge-then-write and TTL | Queues |
| Metrics | An 8-metric dict (depth, in-flight, DLQ, counters, durations) | — |
Async Inference
Enqueue tasks onto KubeMQ Queues and poll the queue-peek result backend for outcomes.
Sync Inference
Blocking request-response inference over KubeMQ Queries with query_task_sync.
Progress Tracking
Stream real-time task progress as KubeMQ Events during long-running inference.
Cancellation
Soft-cancel an in-flight task by overwriting its result with a CANCELLED status.
Metrics at a glance
get_metrics_sync() returns a single dict with eight entries — three live gauges read from KubeMQ
on each call, plus in-memory counters and a histogram. See the
metrics guide for collection patterns and the
API reference for the full schema.
| Metric | Type | Reports |
|---|---|---|
queue_depth | Gauge | Waiting messages in the task queue (the autoscaling signal) |
in_flight | Gauge | Tasks currently being processed |
dlq_depth | Gauge | Waiting messages in the dead-letter queue |
tasks_enqueued_total | Counter | Tasks enqueued since startup |
tasks_completed_total | Counter | Completed counts by status (SUCCESS / FAILURE) |
task_processing_duration_seconds | Histogram | min / max / avg / count of handler durations |
result_storage_retries_total | Counter | Result-storage retry attempts |
consumer_poll_latency_seconds | Gauge | Duration of the last queue poll |
How it compares to the Celery adapter
Ray Serve's reference TaskProcessorAdapter is Celery-backed. KubeMQ collapses the broker, result
backend, and scaler into one component.
| Feature | kubemq-rayserve | CeleryTaskProcessorAdapter |
|---|---|---|
| Message broker | KubeMQ (Kubernetes-native) | Redis / RabbitMQ |
| Sync inference | Built-in (query_task_sync) | Not available |
| Progress tracking | Built-in (report_progress) | Requires custom signals |
| Autoscaling policy | kubemq_queue_depth_policy | Manual HPA configuration |
| DLQ monitoring | on_dlq callback | Requires Celery signals + custom code |
| Result backend | Queue-peek (no extra infra) | Requires separate Redis / DB |
| Setup complexity | 1 dependency (KubeMQ) | 2+ dependencies (broker + backend) |
Next steps
Getting Started
Install the adapter, start a broker, and run a first end-to-end async inference task.
Concepts
The TaskProcessorAdapter model, the queue-peek result backend, lifecycle, and DLQ semantics.
Guides
Autoscaling, configuration, connection security, DLQ & retries, and metrics.
Reference
Every KubeMQAdapterConfig field, the adapter API, the autoscaling policy, and metrics.
New to KubeMQ? Start with the KubeMQ Getting Started guide for core concepts.
Was this page helpful?