Architecture

The KubeMQ Google Cloud Pub/Sub connector is an embedded, wire-protocol bridge inside kubemq-server that speaks the genuine Pub/Sub v1 gRPC services on a dedicated gRPC listener (default TCP 8085, the Pub/Sub emulator convention). The connector is opt-in (disabled by default) — enable it with CONNECTORS_GCP_ENABLE=true (Docker) or spec.gcp.enabled: true (Kubernetes). It runs in emulator mode: no authentication, no TLS, insecure gRPC — exactly like Google's local emulator. Any standard Pub/Sub client connects to it by setting one environment variable, PUBSUB_EMULATOR_HOST — no code changes, no library swap, no emulator to install.

Two KubeMQ primitives back the model:

• A topic maps onto a native KubeMQ Events Store log gcp.{topic} — the authoritative, cross-protocol, replayable source of truth.
• A subscription maps onto a native KubeMQ Queue channel gcp.sub.{subscription} — one filtered copy per subscription.

A publish is written once to the topic log, then fanned out to one Queue copy per subscription. The connector touches the Events Store and Queue primitives only — it does not map onto KubeMQ's commands / queries / events RPC patterns, and there is no request/reply anywhere.

The gRPC emulator listener

A single gRPC server on Connectors.Gcp.Port (default 8085) implements the real Pub/Sub v1 wire protocol — 38 RPCs across four services. Each request passes a fixed three-stage interceptor chain — Recovery → Logger → Traffic-gate — and there is no auth interceptor (emulator mode):

The Traffic-gate interceptor short-circuits requests with transient UNAVAILABLE while the broker is not ready; on a not-ready → ready transition the connector drops all in-memory leases (their downstream transactions are dead) and the poller rebuilds. SDKs see the transient UNAVAILABLE and retry. A nil auth seam is reserved so a future release can add token validation without touching handlers. See Capabilities for the full RPC matrix.

How topics & subscriptions map to KubeMQ

A request arrives at the gRPC listener; the handler parses the resource, dispatches to the matching service, and lands on a KubeMQ primitive. A Publish writes once to the topic's Events Store log and fans out one Queue copy per bound subscription, applying each subscription's filter.

A publish writes once to the Events Store log gcp.{topic} and fans out one filtered Queue copy per subscription on gcp.sub.{subscription}, all backed by the message broker. Snapshots and schemas are registry records with no channel.

The channel mapping is the single most important mental model:

See Channel mapping for the full grammar and Configuration for the connector knobs.

Publish once, then fan out

A Publish is the heart of the model and follows a strict order:

1. Validate the whole batch before enqueuing anything (atomicity): batch size 1..1000; per message — total ≤ 10 MiB, ≤ 100 attributes, attribute key ≤ 256 B (no goog prefix), attribute value ≤ 1024 B, ordering key ≤ 1024 B, and data or attributes non-empty. Any failure rejects the entire batch with INVALID_ARGUMENT and nothing is published.
2. Schema enforcement — if the topic references a schema, every message is validated against it; the whole batch is rejected on the first non-conforming message.
3. Topic-log write — each message is written once to the Events Store log gcp.{topic} (Array.SendEventsStore), assigning a server message id and publish time. This single record is the authoritative, cross-protocol, replayable copy and the source for Seek.
4. Fan-out — one Queue copy per subscription (Array.SendQueueMessage), applying each subscription's filter; a filtered-out message is never enqueued. Detached subscriptions are skipped.

Delivery is then driven from each subscription's queue: every delivered message gets an opaque ack_id under an ack-deadline lease; a 250 ms sweeper expires overdue leases, applies retry backoff, and redelivers — or dead-letters once the receive count exceeds the policy. See Publishing and Subscribing.

Reserved tags & project segment

A PubsubMessage becomes a KubeMQ message whose body is data and whose tags are the message attributes plus three reserved tags carried across the wire:

Native KubeMQ consumers see these tags; they are stripped from attributes when a message is delivered back to a Pub/Sub client. The {p} project segment is parsed and validated but ignored (single-tenant), and the sub. topic-id prefix is reserved.

Cross-protocol interop

Because every topic is a normal KubeMQ Events Store log, a Pub/Sub Publish to topic orders (written to gcp.orders) is consumable by a native KubeMQ Events Store subscriber on the same channel — and the native side sees the three reserved _pubsub_* tags that are stripped for Pub/Sub clients. This lets you migrate one side at a time, or run Pub/Sub-SDK producers alongside native KubeMQ consumers.

A Pub/Sub publish lands on the Events Store log gcp.orders; a native KubeMQ Events Store subscriber reads the same record, including the reserved _pubsub_* tags.

Related