Performance Tuning

kubemq-celery runs Celery tasks over KubeMQ Queues on a gRPC (HTTP/2) connection, which already gives you fewer connections and built-in keepalive compared with Redis or RabbitMQ. This guide covers how to push throughput and latency further by tuning worker concurrency, prefetch, batch receive, and keepalive — and ships three workload profiles you can copy. The exact type and default of every option mentioned here lives in Transport Options.

Prerequisites

• A working Celery app already using kubemq-celery (see Configuration)
• A KubeMQ broker to tune against, ideally under representative load so latency/throughput changes are measurable

Connection efficiency

Because gRPC multiplexes many logical streams over a single HTTP/2 connection, a KubeMQ worker needs far fewer sockets than a Redis or AMQP worker:

The acknowledgment model also affects throughput: with acks_early (the default) the message is auto-acked on receive with zero overhead, while acks_late adds a single gRPC ack() per message after the task succeeds. There is no visibility-timeout race to tune, unlike Redis or SQS — see Concepts for the model.

Recommended baseline settings

A solid starting point for most deployments:

app.conf.update(
    worker_prefetch_multiplier=4,     # 4x concurrency
    worker_concurrency=4,             # match CPU cores
    broker_transport_options={
        "wait_timeout": 1,            # 1s receive poll (default)
        "max_batch_size": 10,         # batch receive for throughput
        "message_expiration": 3600,   # 1h default TTL
    },
)

Tuning guidelines

worker_prefetch_multiplier

Controls how many messages each worker prefetches (buffered in the worker). With max_batch_size > 1, the effective in-flight count is prefetch_multiplier × concurrency.

worker_concurrency

Controls the number of concurrent worker processes or threads.

max_batch_size

How many messages are fetched per gRPC call. Reduces round-trips at the cost of memory.

wait_timeout

How long the receive call blocks waiting for messages.

gRPC keepalive

Keepalive prevents idle connections from being dropped by load balancers or firewalls:

app.conf.broker_transport_options = {
    "grpc_keepalive_time": 30,          # ping every 30s
    "grpc_keepalive_timeout": 10,       # wait 10s for a response
    "grpc_permit_without_calls": True,  # keepalive even when idle
}

Workload profiles

app.conf.update(
    worker_concurrency=8,
    worker_prefetch_multiplier=1,
    broker_transport_options={
        "wait_timeout": 1,
        "max_batch_size": 1,       # minimize latency
    },
)

app.conf.update(
    worker_concurrency=4,
    worker_prefetch_multiplier=8,
    broker_transport_options={
        "wait_timeout": 1,
        "max_batch_size": 50,      # maximize throughput
        "message_expiration": 86400,
    },
)

app.conf.update(
    worker_concurrency=4,
    worker_prefetch_multiplier=4,
    task_routes={
        "myapp.tasks.critical_*": {"queue": "high-priority"},
        "myapp.tasks.batch_*": {"queue": "low-priority"},
    },
    broker_transport_options={
        "wait_timeout": 1,
        "max_batch_size": 10,
        "message_expiration": 3600,
    },
)

# Run separate workers per queue:
# celery -A myapp worker -Q high-priority --concurrency=4 --prefetch-multiplier=1
# celery -A myapp worker -Q low-priority  --concurrency=2 --prefetch-multiplier=8

Benchmarking

The repository ships a benchmark script that measures dispatch rate and round-trip latency against your own broker:

# Start a worker
celery -A benchmark worker --loglevel=info

# Run the benchmark (default: 100 tasks)
python examples/advanced_patterns/benchmark.py

# Custom broker and task count
python examples/advanced_patterns/benchmark.py --broker kubemq://kubemq:50000 --tasks 1000

It reports dispatch rate (messages sent per second) and round-trip latency (dispatch to result retrieval, at p50/p95/p99).

Monitoring performance

Two dashboards give you complementary views:

• KubeMQ Management API dashboard (port 8080; the shared HTTP server on 9090 serves REST/health) — broker-level metrics: queue depth, send/receive rates, and per-channel statistics.
• Flower — task-level metrics: execution times, success/failure rates, worker utilization.

celery -A myapp flower --broker=kubemq://localhost:50000

Celery's built-in inspect commands round out live worker state:

celery -A myapp inspect stats     # uptime, task counts, prefetch
celery -A myapp inspect active    # currently running tasks
celery -A myapp inspect reserved  # prefetched tasks

Related