# Performance Tuning (/integrations/celery/how-to/performance)
`kubemq-celery` runs Celery tasks over [KubeMQ Queues](/learn/queues) on a gRPC (HTTP/2) connection, which already gives you fewer connections and built-in keepalive compared with Redis or RabbitMQ. This guide covers how to push throughput and latency further by tuning worker concurrency, prefetch, batch receive, and keepalive — and ships three workload profiles you can copy. The exact type and default of every option mentioned here lives in [Transport Options](/integrations/celery/reference/transport-options).
## Prerequisites [#prerequisites]
* A working Celery app already using `kubemq-celery` (see [Configuration](/integrations/celery/how-to/configuration))
* A KubeMQ broker to tune against, ideally under representative load so latency/throughput changes are measurable
## Connection efficiency [#connection-efficiency]
Because gRPC multiplexes many logical streams over a single HTTP/2 connection, a KubeMQ worker needs far fewer sockets than a Redis or AMQP worker:
| Aspect | KubeMQ (gRPC) | Redis | RabbitMQ (AMQP) |
| ---------------------- | ------------------------------- | -------------------------------- | -------------------------- |
| Connections per worker | 2–3 (queues + pubsub + backend) | 6–8 (broker + backend + results) | 4–6 (channels + heartbeat) |
| Protocol | HTTP/2 multiplexed | TCP per connection | TCP per channel |
| Keepalive | Built-in gRPC keepalive | Manual PING/PONG | AMQP heartbeats |
| TLS overhead | Single handshake, multiplexed | Per-connection handshake | Per-connection handshake |
The acknowledgment model also affects throughput: with `acks_early` (the default) the message is auto-acked on receive with zero overhead, while `acks_late` adds a single gRPC `ack()` per message after the task succeeds. There is no visibility-timeout race to tune, unlike Redis or SQS — see [Concepts](/integrations/celery/concepts#acknowledgment-model) for the model.
## Recommended baseline settings [#recommended-baseline-settings]
A solid starting point for most deployments:
```python title="recommended.py"
app.conf.update(
worker_prefetch_multiplier=4, # 4x concurrency
worker_concurrency=4, # match CPU cores
broker_transport_options={
"wait_timeout": 1, # 1s receive poll (default)
"max_batch_size": 10, # batch receive for throughput
"message_expiration": 3600, # 1h default TTL
},
)
```
## Tuning guidelines [#tuning-guidelines]
### worker\_prefetch\_multiplier [#worker_prefetch_multiplier]
Controls how many messages each worker prefetches (buffered in the worker). With `max_batch_size > 1`, the effective in-flight count is `prefetch_multiplier × concurrency`.
| Workload | Recommended value | Reason |
| ------------------- | ----------------- | ------------------------------------------- |
| CPU-bound | `1` | Minimize buffering, maximize responsiveness |
| I/O-bound | `4` (default) | Pipeline effect while waiting for I/O |
| High-throughput I/O | `8`–`16` | Maximize worker utilization |
### worker\_concurrency [#worker_concurrency]
Controls the number of concurrent worker processes or threads.
| Workload | Recommended value | Reason |
| --------- | ------------------- | ----------------------- |
| CPU-bound | Number of CPU cores | Avoid oversubscription |
| I/O-bound | 2–4× CPU cores | Overlap I/O waits |
| Mixed | CPU cores + 2 | Balance compute and I/O |
### max\_batch\_size [#max_batch_size]
How many messages are fetched per gRPC call. Reduces round-trips at the cost of memory.
| Setting | Behavior |
| --------------- | ------------------------------------------------------ |
| `1` | One message per gRPC call (lowest latency per message) |
| `10` (default) | Good balance of throughput and memory |
| `20`–`50` | High-throughput workloads |
| `100` (maximum) | Maximum throughput, higher memory usage |
### wait\_timeout [#wait_timeout]
How long the receive call blocks waiting for messages.
| Setting | Behavior |
| ------------- | ------------------------------------------------ |
| `0` | Non-blocking (higher CPU, lowest latency) |
| `1` (default) | 1-second block (good balance) |
| `5`–`10` | Lower CPU usage, slower response to new messages |
`wait_timeout` must be less than Celery's `drain_events` timeout (default 2s) to avoid transport deadlocks. Keep it strictly below.
### gRPC keepalive [#grpc-keepalive]
Keepalive prevents idle connections from being dropped by load balancers or firewalls:
```python title="keepalive.py"
app.conf.broker_transport_options = {
"grpc_keepalive_time": 30, # ping every 30s
"grpc_keepalive_timeout": 10, # wait 10s for a response
"grpc_permit_without_calls": True, # keepalive even when idle
}
```
## Workload profiles [#workload-profiles]
Tasks dispatched from web requests that need fast execution. Raise concurrency, keep prefetch at `1` so no worker hoards queued tasks, and pull one message at a time for the lowest dispatch latency.
```python title="profile_api.py"
app.conf.update(
worker_concurrency=8,
worker_prefetch_multiplier=1,
broker_transport_options={
"wait_timeout": 1,
"max_batch_size": 1, # minimize latency
},
)
```
Processing large volumes of tasks where throughput matters more than latency. Increase prefetch and batch receive to cut gRPC round-trips.
```python title="profile_batch.py"
app.conf.update(
worker_concurrency=4,
worker_prefetch_multiplier=8,
broker_transport_options={
"wait_timeout": 1,
"max_batch_size": 50, # maximize throughput
"message_expiration": 86400,
},
)
```
Different settings per queue using task routing. Run separate workers per queue with their own concurrency and prefetch.
```python title="profile_mixed.py"
app.conf.update(
worker_concurrency=4,
worker_prefetch_multiplier=4,
task_routes={
"myapp.tasks.critical_*": {"queue": "high-priority"},
"myapp.tasks.batch_*": {"queue": "low-priority"},
},
broker_transport_options={
"wait_timeout": 1,
"max_batch_size": 10,
"message_expiration": 3600,
},
)
# Run separate workers per queue:
# celery -A myapp worker -Q high-priority --concurrency=4 --prefetch-multiplier=1
# celery -A myapp worker -Q low-priority --concurrency=2 --prefetch-multiplier=8
```
## Benchmarking [#benchmarking]
The repository ships a benchmark script that measures dispatch rate and round-trip latency against your own broker:
```bash
# Start a worker
celery -A benchmark worker --loglevel=info
# Run the benchmark (default: 100 tasks)
python examples/advanced_patterns/benchmark.py
# Custom broker and task count
python examples/advanced_patterns/benchmark.py --broker kubemq://kubemq:50000 --tasks 1000
```
It reports **dispatch rate** (messages sent per second) and **round-trip latency** (dispatch to result retrieval, at p50/p95/p99).
## Monitoring performance [#monitoring-performance]
Two dashboards give you complementary views:
* **KubeMQ Management API dashboard** (port `8080`; the shared HTTP server on `9090` serves REST/health) — broker-level metrics: queue depth, send/receive rates, and per-channel statistics.
* **Flower** — task-level metrics: execution times, success/failure rates, worker utilization.
```bash
celery -A myapp flower --broker=kubemq://localhost:50000
```
Celery's built-in inspect commands round out live worker state:
```bash
celery -A myapp inspect stats # uptime, task counts, prefetch
celery -A myapp inspect active # currently running tasks
celery -A myapp inspect reserved # prefetched tasks
```
## Related [#related]