Kafka Streams
Run Kafka Streams applications against KubeMQ — settings, internal topics, exactly-once, and what has been proven.
A Kafka Streams application runs against KubeMQ by changing bootstrap.servers. Streams creates its
internal topics — repartition topics and compacted changelog topics for state stores — through the
same admin calls it uses against Apache Kafka.
Settings
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "orders-aggregator");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kubemq-next-kafka.kubemq.svc:9092");
// -1 (the Streams default since 3.0) uses the server's default, which on KubeMQ is every node.
// Any explicit value up to the node count is accepted from next v1.2.0.
props.put(StreamsConfig.REPLICATION_FACTOR_CONFIG, -1);Changelog topics are compacted; compaction runs on the next storage engine the Kafka connector
always uses. A state store restored from its changelog after a restart comes back with the values it
had.
Exactly-once
processing.guarantee=exactly_once_v2 rests on transactions whose consumer offsets are committed
with the consumer group's metadata. That protocol is exercised end to end by the repository's
application simulator — a consume-transform-produce pipeline in Java and in Python, with deliberate
aborts, on a three-node cluster, including a run where the cluster leader is killed mid-traffic —
and every order was committed exactly once with no aborted output visible. Kafka Streams itself has
not yet been run with exactly_once_v2; run your own topology's exactly-once test before relying on
it.
The next-generation transaction protocol (Kafka 4.0's transaction version 2) is not served; clients fall back to the protocol above, as they do against an Apache Kafka cluster that has not enabled it.
Authorization
With authorization on, the application's principal needs Read on its input topics, Write on
its output topics and on every internal topic (they are named <application.id>-…), and Write
on the consumer group named after application.id. See
Translating Kafka ACLs.
What has been proven
A stateless topology (filter and map) and a stateful one (group-by-key count over a compacted changelog, with a queryable state store) were run with Kafka Streams 3.9.1 against a three-node KubeMQ cluster and produced the correct results. The Fitness Matrix carries the proof tier. ksqlDB is a separate runtime and is out of scope.
Was this page helpful?
Kafka Connect
Run Kafka Connect workers against KubeMQ — the worker settings that matter, the internal topics it creates, and what has been proven.
Schema Registry
Use Confluent Schema Registry with KubeMQ — serializers work unchanged, and the registry service itself runs against KubeMQ as its storage.