Reliability
Delivery guarantees over KubeMQ — dead-letter topics, retry backoff, at-least-once redelivery, node-local exactly-once, and retention clamped to the broker.
This guide covers the connector's delivery guarantees: the at-least-once baseline, retry backoff and dead-letter topics, exactly-once delivery and its node-local boundary, and how retention is clamped to the message broker. Several behaviors carry node-local caveats — read the callouts.
At-least-once is the baseline
Every subscription delivers at least once. A delivered message is held under an ack-deadline
lease; if it is not acked before the deadline, a 250 ms sweeper applies the retry backoff and
redelivers it (the receive count increments). An explicit nack — ModifyAckDeadline(0) —
redelivers immediately. On a broker not-ready → ready transition the connector drops all in-memory
leases and redelivers any in-flight messages.
Design consumers to be idempotent. Without enable_exactly_once_delivery, a message may be
delivered more than once — on deadline expiry, nack, or broker recovery. Exactly-once tightens
this, but only within a single node (see below). A message is never silently lost on a graceful
path; it is redelivered and the receive count increments.
Retry backoff
Ack-deadline expiry redelivers with an exponential backoff clamped to the subscription's
[min, max] retry policy (defaults 10 s … 600 s), so a transiently failing message gets spaced-out
retries rather than a hot loop. An explicit nack (ModifyAckDeadline(0)) bypasses the backoff
and redelivers immediately. Ordering keys keep their in-order guarantee across redeliveries — an
ordered message redelivers before any later message for the same key.
Dead-letter topics
A dead-letter topic is the connector's poison-message safety valve: a message that keeps failing is moved out of the subscription instead of redelivering forever.
- Create a subscription with a
dead_letter_topicand amax_delivery_attemptsvalue. - Each delivery increments the message's receive count; a nack or an expired lease drives redelivery with backoff.
- When the receive count exceeds
max_delivery_attempts, the 250 ms sweeper republishes the message to the dead-letter topic (a connector-level fan-out through the normal publish path) and acks the original — so it leaves the source subscription.
The dead-letter topic is an ordinary Pub/Sub topic backed by its own Events Store log gcp.{dlt},
so you attach a subscription to it and consume the failed messages like any other.
max_delivery_attempts must be 5..100. A value of 0 means unset — no dead-lettering, so
the message redelivers indefinitely. Any non-zero value must be in the range 5..100; outside
that range the subscription create is rejected with INVALID_ARGUMENT. This is Google's own rule,
enforced up front. See Limits & rules.
Push subscriptions share the same retry → dead-letter pipeline (retry on non-2xx / timeout, dead-letter on exhaustion). See Push delivery.
Exactly-once delivery
A subscription created with enable_exactly_once_delivery strengthens the contract: once a message
is successfully acknowledged, it will not be redelivered, and the ack itself is confirmed so
the client knows it took effect. The ack contract changes so the SDK can resolve each result:
- StreamingPull returns an
AcknowledgeConfirmation/ModifyAckDeadlineConfirmation. Expired or unknownack_ids appear ininvalid_ack_ids; transient broker failures appear intemporary_failed_ack_ids(the client retries those). - Unary
Acknowledge/ModifyAckDeadlineon an invalid id returns aFAILED_PRECONDITIONstatus carrying anErrorInfo{reason: PERMANENT_FAILURE_INVALID_ACK_ID}.
The unary invalid-ack error is FAILED_PRECONDITION + ErrorInfo, not INVALID_ARGUMENT.
This matches the real Google SDK contract — the client library reads the ack result from the
ErrorInfo reason. A naive INVALID_ARGUMENT would break that resolution. See
Error codes.
Exactly-once is node-local
Exactly-once is node-local — pin StreamingPull to one node. An ack_id is a token that carries
the node id of the node that minted it. An ack_id minted on one node is invalid on another
(the node id won't match), so there is no cluster-wide distributed exactly-once. In a KubeMQ
cluster you must either pin an exactly-once subscription's StreamingPull traffic to a single
node (a sticky load balancer with session affinity), or accept at-least-once across nodes.
Single-node deployments are unaffected. This is the most important caveat in this connector.
Retention is clamped to the broker
Per-resource retention is clamped to the message broker's global Store.MaxRetention ceiling.
GetTopic / GetSubscription echo the requested retention value, but fan-out, seek, and the
dashboard use the effective (clamped) value. Because the replayable topic log is the source for
Seek, what is actually retained — and therefore how far back you can rewind — depends on the
broker ceiling, not just the value you requested.
Node-local state summary (cluster)
Three pieces of delivery state are node-local. Topic / subscription / snapshot / schema
records are synchronized cluster-wide, so resource existence is cluster-wide. But these
delivery-state pieces are node-local and require a sticky load balancer in a cluster:
exactly-once ack_ids (node id baked into the token); StreamingPull leases, in-flight tracking, and
flow-control counters; and push-delivery workers with their in-flight retries. Single-node
deployments are unaffected.
Error quick reference
| Trigger | Result |
|---|---|
| Ack deadline expires without ack | redelivery; receive count increments |
ModifyAckDeadline(0) | immediate nack / redeliver |
Receive count exceeds max_delivery_attempts (DLQ set) | republished to dead_letter_topic, original acked |
| Exactly-once unary ack of an expired / unknown id | FAILED_PRECONDITION + ErrorInfo(PERMANENT_FAILURE_INVALID_ACK_ID) |
| Exactly-once StreamingPull ack of an expired id | id appears in invalid_ack_ids |
max_delivery_attempts outside 5..100 | rejected (INVALID_ARGUMENT) |
| Broker not-ready → ready transition | in-memory leases dropped; in-flight messages redelivered |
Related
Subscribing
The lease model, ack / nack / extend, flow control, and exactly-once on the pull paths.
Push delivery
The per-subscription delivery worker and its shared retry → dead-letter pipeline.
Migrating from Google Cloud Pub/Sub
The deviations to weigh — exactly-once scope, retention clamping, and node-local state.
Was this page helpful?
Push Delivery
Push subscriptions over KubeMQ — the per-subscription delivery worker, wrapped JSON envelope vs no_wrapper, 2xx-acks, retry with backoff, and dead-letter.
Schema Validation
Avro and Protobuf schema enforcement over KubeMQ — CreateSchema, topic schema_settings, enforce-on-publish, ≤300 KB definitions, and revisions.