# Ray Serve (/integrations/rayserve)



`kubemq-rayserve` is a Python package that plugs [KubeMQ](https://kubemq.io/) into
[Ray Serve](https://docs.ray.io/en/latest/serve/index.html) as a task backend. It provides
`KubeMQTaskProcessorAdapter` — the first non-Celery adapter for Ray Serve's
`TaskProcessorAdapter` framework — so a Ray Serve deployment can run **queue-based asynchronous**
and **blocking synchronous** ML inference, with KubeMQ acting as the message broker, the result
store, the autoscaling signal, and the progress event bus all at once.

The adapter is a native gRPC SDK client. It connects to a KubeMQ broker on port `50000` and is
always available — there is no connector to enable on the server. You install one package, point
it at a broker, and Ray Serve drives the rest of the lifecycle.

Ray Serve is an [integration](/integrations#what-an-integration-is): a client-side library that
embeds a KubeMQ SDK, distinct from the server-side [connectors](/connectors) (MCP, A2A,
CloudEvents) that run inside the broker.

## Why KubeMQ + Ray Serve [#why-kubemq--ray-serve]

* **One Kubernetes-native broker** — a single binary carries the task queue, the synchronous query
  channel, the result store, and the progress event stream. No Erlang, Redis, or RabbitMQ to
  operate alongside Ray.
* **Built-in sync inference** — `query_task_sync` gives blocking request-response inference that the
  reference Celery adapter does not offer.
* **Built-in progress tracking** — `report_progress` streams real-time task progress over KubeMQ
  Events; Celery requires custom signals.
* **Queue-peek result backend** — results live in KubeMQ itself, with no separate Redis or database
  to provision.
* **Native autoscaling** — `kubemq_queue_depth_policy` scales Ray Serve replicas from live queue
  depth, and the same depth drives [KEDA](/integrations/keda) for cluster-level scaling.
* **One-dependency setup** — only KubeMQ is required, versus a broker plus a separate result
  backend.

## Install [#install]

```bash title="terminal"
uv pip install kubemq-rayserve
```

| Requirement              | Version   |
| ------------------------ | --------- |
| Python                   | >= 3.10   |
| Ray Serve (`ray[serve]`) | >= 2.50.0 |
| `kubemq` (Python SDK)    | >= 4.1.5  |
| `pydantic`               | >= 2.0    |

A running KubeMQ broker is required (default address `localhost:50000`). Start one locally with
Docker:

<RunKubeMQ ports="[50000, 9090]" />

<Callout type="info">
  Port `50000` is KubeMQ's gRPC endpoint — the only port the adapter's SDK clients use. Port `9090`
  is the shared HTTP server (REST and the AI-agent connectors); the dashboard runs separately on
  port `8080`. The adapter and the `kubemq_queue_depth_policy` autoscaler both speak gRPC on `50000`.
</Callout>

## Architecture [#architecture]

A producer enqueues a serialized task onto a KubeMQ Queue. Inside the Ray Serve deployment, the
`KubeMQTaskProcessorAdapter` runs a consumer thread that polls the queue, dispatches each task to a
registered handler, and writes the result back into the queue-peek result backend. A parallel Query
channel serves blocking sync inference, and an Events channel carries progress updates — all over
the same gRPC connection to port `50000`.

<Mermaid
  chart="`
graph LR
PROD[&#x22;Producer<br/>(HTTP / gRPC client)&#x22;]
subgraph RAY[&#x22;Ray Serve deployment&#x22;]
  ADP{{&#x22;KubeMQTaskProcessorAdapter&#x22;}}
end
Q[&#x22;Queue<br/>tasks + results&#x22;]
CQ[&#x22;Query<br/>sync inference&#x22;]
EV[&#x22;Events<br/>{queue}.progress&#x22;]
BROKER[&#x22;KubeMQ broker<br/>gRPC :50000&#x22;]

PROD -- &#x22;enqueue_task / query_task_sync&#x22; --> ADP
ADP -- &#x22;Queues SDK&#x22; --> Q
ADP -- &#x22;CQ SDK&#x22; --> CQ
ADP -- &#x22;PubSub SDK&#x22; --> EV
Q --> BROKER
CQ --> BROKER
EV --> BROKER

class PROD client
class ADP aiway
class Q,CQ,EV,BROKER broker
`"
/>

*The adapter maps Ray Serve's task model onto three KubeMQ primitives — Queues, Queries, and Events — over one gRPC connection.*

The adapter never re-teaches KubeMQ messaging: async tasks ride [Queues](/learn/queues), sync
inference rides request-response [RPC](/learn/rpc) (Queries), and progress rides
[Events](/learn/events). Each capability page documents the adapter API and links to the underlying
concept.

## Capabilities [#capabilities]

`KubeMQTaskProcessorAdapter` exposes eight capabilities, grouped into three inference models plus
the operational features that surround them.

| Capability        | What it does                                                       | KubeMQ primitive            |
| ----------------- | ------------------------------------------------------------------ | --------------------------- |
| Async inference   | Enqueue a task, poll the result backend for the outcome            | [Queues](/learn/queues)     |
| Sync inference    | Blocking request-response inference via `query_task_sync`          | [RPC](/learn/rpc) (Queries) |
| Progress tracking | Stream live `report_progress` updates to subscribers               | [Events](/learn/events)     |
| Cancellation      | Soft-cancel a task — overwrite its result with `CANCELLED`         | Queues                      |
| DLQ monitoring    | `on_dlq` callback alerts on permanently failed tasks               | Queues                      |
| Autoscaling       | Scale replicas on queue depth (`kubemq_queue_depth_policy` / KEDA) | Queues                      |
| Result backend    | Queue-peek result storage with purge-then-write and TTL            | Queues                      |
| Metrics           | An 8-metric dict (depth, in-flight, DLQ, counters, durations)      | —                           |

<Cards>
  <Card title="Async Inference" href="/integrations/rayserve/how-to/async-inference" description="Enqueue tasks onto KubeMQ Queues and poll the queue-peek result backend for outcomes." />

  <Card title="Sync Inference" href="/integrations/rayserve/how-to/sync-inference" description="Blocking request-response inference over KubeMQ Queries with query_task_sync." />

  <Card title="Progress Tracking" href="/integrations/rayserve/how-to/progress-tracking" description="Stream real-time task progress as KubeMQ Events during long-running inference." />

  <Card title="Cancellation" href="/integrations/rayserve/how-to/cancellation" description="Soft-cancel an in-flight task by overwriting its result with a CANCELLED status." />
</Cards>

## Metrics at a glance [#metrics-at-a-glance]

`get_metrics_sync()` returns a single dict with eight entries — three live gauges read from KubeMQ
on each call, plus in-memory counters and a histogram. See the
[metrics guide](/integrations/rayserve/how-to/metrics) for collection patterns and the
[API reference](/integrations/rayserve/reference/api#metrics) for the full schema.

| Metric                             | Type      | Reports                                                     |
| ---------------------------------- | --------- | ----------------------------------------------------------- |
| `queue_depth`                      | Gauge     | Waiting messages in the task queue (the autoscaling signal) |
| `in_flight`                        | Gauge     | Tasks currently being processed                             |
| `dlq_depth`                        | Gauge     | Waiting messages in the dead-letter queue                   |
| `tasks_enqueued_total`             | Counter   | Tasks enqueued since startup                                |
| `tasks_completed_total`            | Counter   | Completed counts by status (`SUCCESS` / `FAILURE`)          |
| `task_processing_duration_seconds` | Histogram | `min` / `max` / `avg` / `count` of handler durations        |
| `result_storage_retries_total`     | Counter   | Result-storage retry attempts                               |
| `consumer_poll_latency_seconds`    | Gauge     | Duration of the last queue poll                             |

## How it compares to the Celery adapter [#how-it-compares-to-the-celery-adapter]

Ray Serve's reference `TaskProcessorAdapter` is Celery-backed. KubeMQ collapses the broker, result
backend, and scaler into one component.

| Feature            | kubemq-rayserve              | CeleryTaskProcessorAdapter            |
| ------------------ | ---------------------------- | ------------------------------------- |
| Message broker     | KubeMQ (Kubernetes-native)   | Redis / RabbitMQ                      |
| Sync inference     | Built-in (`query_task_sync`) | Not available                         |
| Progress tracking  | Built-in (`report_progress`) | Requires custom signals               |
| Autoscaling policy | `kubemq_queue_depth_policy`  | Manual HPA configuration              |
| DLQ monitoring     | `on_dlq` callback            | Requires Celery signals + custom code |
| Result backend     | Queue-peek (no extra infra)  | Requires separate Redis / DB          |
| Setup complexity   | 1 dependency (KubeMQ)        | 2+ dependencies (broker + backend)    |

## Next steps [#next-steps]

<Cards>
  <Card title="Getting Started" href="/integrations/rayserve/tutorials/getting-started" description="Install the adapter, start a broker, and run a first end-to-end async inference task." />

  <Card title="Concepts" href="/integrations/rayserve/concepts" description="The TaskProcessorAdapter model, the queue-peek result backend, lifecycle, and DLQ semantics." />

  <Card title="Guides" href="/integrations/rayserve/how-to/autoscaling" description="Autoscaling, configuration, connection security, DLQ & retries, and metrics." />

  <Card title="Reference" href="/integrations/rayserve/reference/configuration" description="Every KubeMQAdapterConfig field, the adapter API, the autoscaling policy, and metrics." />
</Cards>

New to KubeMQ? Start with the [KubeMQ Getting Started guide](/deploy) for core
concepts.