# Reliability (/connectors/gcp-pub-sub/how-to/reliability)



This guide covers the connector's delivery guarantees: the **at-least-once** baseline, **retry
backoff** and **dead-letter topics**, **exactly-once** delivery and its node-local boundary, and
how **retention** is clamped to the message broker. Several behaviors carry node-local caveats —
read the callouts.

## At-least-once is the baseline [#at-least-once-is-the-baseline]

Every subscription delivers **at least once**. A delivered message is held under an ack-deadline
lease; if it is not acked before the deadline, a 250 ms sweeper applies the retry backoff and
**redelivers** it (the receive count increments). An explicit nack — `ModifyAckDeadline(0)` —
redelivers immediately. On a broker not-ready → ready transition the connector drops all in-memory
leases and redelivers any in-flight messages.

<Callout type="warn">
  **Design consumers to be idempotent.** Without `enable_exactly_once_delivery`, a message may be
  delivered more than once — on deadline expiry, nack, or broker recovery. Exactly-once tightens
  this, but only within a single node (see below). A message is never silently lost on a graceful
  path; it is **redelivered** and the receive count increments.
</Callout>

## Retry backoff [#retry-backoff]

Ack-deadline expiry redelivers with an **exponential backoff** clamped to the subscription's
`[min, max]` retry policy (defaults 10 s … 600 s), so a transiently failing message gets spaced-out
retries rather than a hot loop. An explicit nack (`ModifyAckDeadline(0)`) **bypasses** the backoff
and redelivers immediately. Ordering keys keep their in-order guarantee across redeliveries — an
ordered message redelivers before any later message for the same key.

## Dead-letter topics [#dead-letter-topics]

A dead-letter topic is the connector's poison-message safety valve: a message that keeps failing
is moved out of the subscription instead of redelivering forever.

1. Create a subscription with a `dead_letter_topic` and a `max_delivery_attempts` value.
2. Each delivery increments the message's receive count; a nack or an expired lease drives
   redelivery with backoff.
3. When the &#x2A;*receive count exceeds `max_delivery_attempts`**, the 250 ms sweeper **republishes**
   the message to the dead-letter topic (a connector-level fan-out through the normal publish path)
   and **acks the original** — so it leaves the source subscription.

The dead-letter topic is an ordinary Pub/Sub topic backed by its own Events Store log `gcp.{dlt}`,
so you attach a subscription to it and consume the failed messages like any other.

<Callout type="warn">
  **`max_delivery_attempts` must be 5..100.** A value of `0` means *unset* — no dead-lettering, so
  the message redelivers indefinitely. Any non-zero value **must** be in the range 5..100; outside
  that range the subscription create is rejected with `INVALID_ARGUMENT`. This is Google's own rule,
  enforced up front. See [Limits & rules](/connectors/gcp-pub-sub/reference/limits-and-rules).
</Callout>

Push subscriptions share the same retry → dead-letter pipeline (retry on non-2xx / timeout,
dead-letter on exhaustion). See [Push delivery](/connectors/gcp-pub-sub/how-to/push-delivery).

## Exactly-once delivery [#exactly-once-delivery]

A subscription created with `enable_exactly_once_delivery` strengthens the contract: once a message
is **successfully acknowledged**, it will not be redelivered, and the ack itself is **confirmed** so
the client knows it took effect. The ack contract changes so the SDK can resolve each result:

* **StreamingPull** returns an `AcknowledgeConfirmation` / `ModifyAckDeadlineConfirmation`. Expired
  or unknown `ack_id`s appear in `invalid_ack_ids`; transient broker failures appear in
  `temporary_failed_ack_ids` (the client retries those).
* **Unary** `Acknowledge` / `ModifyAckDeadline` on an invalid id returns a `FAILED_PRECONDITION`
  status carrying an `ErrorInfo{reason: PERMANENT_FAILURE_INVALID_ACK_ID}`.

<Callout type="info">
  **The unary invalid-ack error is `FAILED_PRECONDITION` + `ErrorInfo`, not `INVALID_ARGUMENT`.**
  This matches the real Google SDK contract — the client library reads the ack result from the
  `ErrorInfo` reason. A naive `INVALID_ARGUMENT` would break that resolution. See
  [Error codes](/connectors/gcp-pub-sub/reference/error-codes).
</Callout>

### Exactly-once is node-local [#exactly-once-is-node-local]

<Callout type="warn">
  **Exactly-once is node-local — pin StreamingPull to one node.** An `ack_id` is a token that carries
  the **node id** of the node that minted it. An `ack_id` minted on one node is **invalid on another**
  (the node id won't match), so there is no cluster-wide distributed exactly-once. In a KubeMQ
  **cluster** you must either **pin** an exactly-once subscription's StreamingPull traffic to a single
  node (a sticky load balancer with session affinity), or **accept at-least-once** across nodes.
  Single-node deployments are unaffected. This is the most important caveat in this connector.
</Callout>

## Retention is clamped to the broker [#retention-is-clamped-to-the-broker]

Per-resource retention is **clamped** to the message broker's global `Store.MaxRetention` ceiling.
`GetTopic` / `GetSubscription` echo the requested retention value, but fan-out, seek, and the
dashboard use the &#x2A;*effective (clamped)** value. Because the replayable topic log is the source for
`Seek`, what is actually retained — and therefore how far back you can rewind — depends on the
broker ceiling, not just the value you requested.

## Node-local state summary (cluster) [#node-local-state-summary-cluster]

<Callout type="warn">
  **Three pieces of delivery state are node-local.** Topic / subscription / snapshot / schema
  **records** are synchronized cluster-wide, so resource **existence** is cluster-wide. But these
  delivery-state pieces are node-local and require a **sticky load balancer** in a cluster:
  exactly-once `ack_id`s (node id baked into the token); StreamingPull leases, in-flight tracking, and
  flow-control counters; and push-delivery workers with their in-flight retries. Single-node
  deployments are unaffected.
</Callout>

## Error quick reference [#error-quick-reference]

| Trigger                                                 | Result                                                                |
| ------------------------------------------------------- | --------------------------------------------------------------------- |
| Ack deadline expires without ack                        | redelivery; receive count increments                                  |
| `ModifyAckDeadline(0)`                                  | immediate nack / redeliver                                            |
| Receive count exceeds `max_delivery_attempts` (DLQ set) | republished to `dead_letter_topic`, original acked                    |
| Exactly-once unary ack of an expired / unknown id       | `FAILED_PRECONDITION` + `ErrorInfo(PERMANENT_FAILURE_INVALID_ACK_ID)` |
| Exactly-once StreamingPull ack of an expired id         | id appears in `invalid_ack_ids`                                       |
| `max_delivery_attempts` outside 5..100                  | rejected (`INVALID_ARGUMENT`)                                         |
| Broker not-ready → ready transition                     | in-memory leases dropped; in-flight messages redelivered              |

## Related [#related]

<Cards>
  <Card title="Subscribing" href="/connectors/gcp-pub-sub/how-to/subscribing" description="The lease model, ack / nack / extend, flow control, and exactly-once on the pull paths." />

  <Card title="Push delivery" href="/connectors/gcp-pub-sub/how-to/push-delivery" description="The per-subscription delivery worker and its shared retry → dead-letter pipeline." />

  <Card title="Migrating from Google Cloud Pub/Sub" href="/connectors/gcp-pub-sub/reference/migration-from-gcp" description="The deviations to weigh — exactly-once scope, retention clamping, and node-local state." />
</Cards>