Kafka Connect

Kafka Connect runs unchanged against KubeMQ: point the workers' bootstrap.servers at the Kafka listener and they create their internal topics, form a worker group, and run connectors as they do against Apache Kafka. Connect is a Kafka client like any other; there is nothing KubeMQ-specific to install.

Worker settings

bootstrap.servers=kubemq-next-kafka.kubemq.svc:9092
group.id=connect-cluster

config.storage.topic=connect-configs
offset.storage.topic=connect-offsets
status.storage.topic=connect-status

# Connect's defaults are 3. From next v1.2.0 KubeMQ accepts any value up to its node count —
# keep 3 on a three-node cluster, use 1 on a single node (Apache Kafka refuses 3 on one broker
# too). Next v1.1.0 and earlier accepted only 1.
config.storage.replication.factor=3
offset.storage.replication.factor=3
status.storage.replication.factor=3

key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter

The config and status topics are created compacted (cleanup.policy=compact). Compaction runs on KubeMQ's next storage engine, which is the engine the Kafka connector always runs on, so there is nothing to enable (Compacted topics).

Workers can bootstrap to any node. Topic creation and config changes sent to a follower are forwarded to the cluster leader.

Authorization

With authorization on, the worker's principal needs, as KubeMQ policy rules (Translating Kafka ACLs):

• Write on the three internal topics and on every topic a source connector produces to;
• Read on every topic a sink connector consumes from;
• Write on the worker group (group.id) and on each sink connector's consumer group (connect-<connector-name> by default) — committing a consumer offset needs Write here, not the Group Read Apache Kafka asks for.

What has been proven

Distributed mode was run with real Apache Kafka Connect 3.9.1 workers against a three-node KubeMQ cluster: the internal topics (compacted ones included), connector lifecycle, a two-worker rebalance, a source-to-sink pipeline with no gaps and no duplicates, a worker killed mid-run, and a KubeMQ node killed mid-run with the connectors staying RUNNING. The Fitness Matrix carries the proof tier.

Not yet run: standalone mode and Debezium change-data-capture connectors, and exactly-once source connectors (exactly.once.source.support). Treat those as untested rather than unsupported.