KubeMQ
ConnectorsGoogle Cloud Pub/SubHow-to guides

Subscribing

Consume over KubeMQ — Pull vs StreamingPull with flow control, ack-deadline leases, ModifyAckDeadline nack/extend, and exactly-once with its node-local note.

This guide covers the consume surface: subscription lifecycle, the Pull vs StreamingPull paths, the ack-deadline lease model (ack / nack / extend), flow control, exactly-once delivery, and the periodic server-initiated reconnect. Every subscription is a native KubeMQ Queue channel gcp.sub.{subscription} (see Channel mapping).

Subscription lifecycle

The Subscriber surface ships 16 RPCs (see Capabilities):

  • CreateSubscription — binds to a topic; the queue is created lazily. A filter is compiled at create-time and is immutable thereafter. Export subscriptions (BigQuery / Cloud Storage / Bigtable) and ingestion are rejected (INVALID_ARGUMENT).
  • GetSubscription / ListSubscriptions.
  • UpdateSubscription — a FieldMask over ack deadline, retention, dead-letter, retry, push, exactly-once, and labels. name and filter are immutable.
  • DeleteSubscription — drops the backlog and any leases.
  • Pull, Acknowledge, ModifyAckDeadline, StreamingPull, ModifyPushConfig, Seek, and the five snapshot RPCs.

Pull vs StreamingPull

Both paths read from the subscription's queue channel through a poller and place each delivered message under an ack-deadline lease.

Unary Pull

Pull returns up to max_messages (≤ 1000) currently-available messages, each with an ack_id. You ack with Acknowledge(ack_ids) or nack/extend with ModifyAckDeadline. A Pull on a detached subscription returns FAILED_PRECONDITION.

StreamingPull

StreamingPull is a bidirectional stream: the server pushes messages as they arrive and the client sends back ack_ids, modify_deadline requests, and flow-control settings on the same stream. This is what the high-level subscriber.Receive(...) / subscription.on('message', ...) helpers use.

Leases are subscription-owned, not stream-owned. An ack on one StreamingPull stream correctly resolves a message that was delivered on a different stream (cross-stream ack). This matters for clients that reconnect or run multiple streams.

The ack-deadline lease

Every delivered message gets an opaque ack_id — a base64-JSON token carrying the subscription, channel, node id, broker transaction id, sequence, receive count, lease id, and deadline. The message stays leased (invisible to other consumers) until the deadline:

ActionEffect
Acknowledge(ack_ids) (or StreamingPull ack_ids)Decodes each id and acks the broker sequence — the message is removed.
ModifyAckDeadline(0)Immediate nack / redeliver (bypasses retry backoff).
ModifyAckDeadline(>0)Extends the deadline. Valid range 10..600 s.
Deadline expiryA 250 ms sweeper expires the lease, applies the retry backoff, and redelivers — or dead-letters once the receive count exceeds the policy.

The default ack deadline is CONNECTORS_GCP_DEFAULT_ACK_DEADLINE_SECONDS (default 10 s, range 10..600).

Ack deadline is 0 (nack) or 10..600 s. A value between 1 and 9 is not valid; 0 means nack (immediate redelivery). See Limits & rules.

// Nack immediately to force redelivery, or extend the lease while you work.
sub.ReceiveSettings.MaxOutstandingMessages = 100
err := sub.Receive(ctx, func(ctx context.Context, m *pubsub.Message) {
    if !canProcess(m) {
        m.Nack() // ModifyAckDeadline(0) → immediate redeliver
        return
    }
    m.Ack() // Acknowledge → removed from the queue
})

Flow control

On a StreamingPull stream the client sets max_outstanding_messages / max_outstanding_bytes; the connector keeps per-stream counters keyed by the ack_ids that stream emitted (≤ 0 = use the connector's CONNECTORS_GCP_MAX_OUTSTANDING_MESSAGES ceiling, default 1000). Outstanding count is released on ack / nack / expiry, and fully released on stream disconnect — without waiting for lease expiry.

A separate hard ceiling, CONNECTORS_GCP_MAX_INFLIGHT_PER_SUBSCRIPTION (default 20,000), caps the total leased (un-acked) messages per subscription across all streams. These knobs are in Configuration.

Periodic reconnect

A StreamingPull stream is closed by the server after CONNECTORS_GCP_STREAM_CLOSE_SECONDS (default 1800 s / 30 min) with UNAVAILABLE. This is normal — client libraries transparently reconnect and your receive callback keeps running. It bounds per-stream resource lifetime. Do not treat the periodic UNAVAILABLE as an error. See Connectivity & emulator mode.

Ack-deadline reset on broker recovery

On a broker not-ready → ready transition the connector drops all in-memory leases (their downstream transactions are dead) and the poller rebuilds. Any in-flight messages are redelivered after recovery — design consumers to be idempotent. See Reliability.

Exactly-once delivery

A subscription with enable_exactly_once_delivery changes the ack contract:

  • StreamingPull returns AcknowledgeConfirmation / ModifyAckDeadlineConfirmation messages: expired/unknown ids appear in invalid_ack_ids; transient broker failures in temporary_failed_ack_ids (the client retries those).
  • A unary Acknowledge / ModifyAckDeadline returns FAILED_PRECONDITION with an ErrorInfo(reason: PERMANENT_FAILURE_INVALID_ACK_ID) for an unparseable/expired/unknown id. This matches the real Google SDK contract (the SDK resolves the ack result from the ErrorInfo reason), and differs from a naive INVALID_ARGUMENT.

Exactly-once is node-local. An ack_id minted on one node is invalid on another (the token's node id won't match) — by design (no cross-node distributed exactly-once). In a cluster, pin an exactly-once subscription's StreamingPull traffic to one node with a sticky load balancer, or accept at-least-once across nodes. See Reliability and Error codes.

Error quick reference

TriggerResult
Pull / Seek on a detached subscriptionFAILED_PRECONDITION
Export-subscription / ingestion config on CreateSubscriptionINVALID_ARGUMENT
UpdateSubscription of name or filterrejected (immutable)
Exactly-once unary ack of an expired/unknown idFAILED_PRECONDITION + ErrorInfo(PERMANENT_FAILURE_INVALID_ACK_ID)
Leased messages exceed CONNECTORS_GCP_MAX_INFLIGHT_PER_SUBSCRIPTIONnew deliveries throttled

Was this page helpful?

On this page