Migrate from Kafka

Drop-in level: endpoint-only. KubeMQ speaks the native Kafka wire protocol — real librdkafka/kcat/Java clients connect unchanged; you repoint bootstrap.servers, with no client-library swap or code change. Moving an existing Apache Kafka, Amazon MSK, or Confluent cluster onto KubeMQ is more than pointing at a new endpoint if you want consumers to resume without reprocessing history — that historical topic-data and offset move is what the separate kmq migrate tool below handles. This guide covers the read-only fitness check, the four-phase kmq migrate tool, per-source auth, the large-history MirrorMaker 2 hybrid, rollback, and a post-cutover smoke test.

Overview

Migrating an existing Kafka workload onto KubeMQ has two stages: assess, then migrate. kmq assess kafka scans your source cluster read-only and reports, per topic, whether it's a straight repoint or needs a workaround. kmq migrate then does the actual work — copying topic history and consumer-group offsets to KubeMQ in four phases (assess → replicate → translate → cutover) — so consumers resume where they left off instead of reprocessing everything from scratch.

Storage engine (zero-config)

Kafka on KubeMQ requires the next storage engine — but you don't set it manually. On a fresh deployment, enabling the Kafka connector with the engine left unset auto-selects next and logs a NOTICE; there is no separate "set store.engine=next first" step.

Pinning STORE_ENGINE=next (or store.engine: next in config.yaml) always works and always wins — it skips the auto-select probe entirely, which is the predictable choice for IaC/GitOps. See Storage Engines for the full engine model, mode isolation, and durability guarantees.

Assess fit — kmq assess kafka

Before touching anything, run the read-only assessor against your source cluster:

kmq assess kafka --bootstrap your-broker:9092 [--tls --sasl-mechanism scram-sha-256 --sasl-username <user> --sasl-password <pass>]

kmq assess kafka never produces, commits an offset, or creates a topic — it's safe to run against a production cluster. It reports, per topic, one of four verdicts, plus a single overall migratable verdict:

Hard BLOCKED:

• Compacted topics (cleanup.policy=compact) — compaction leaves non-contiguous offsets (compaction holes) that the exact offset-translation path can't follow. Move these via bulk data copy (the MirrorMaker 2 hybrid below) and rebuild consumer state, or keep them on the source.
• More than 256 partitions on a topic — KubeMQ's per-topic partition cap. There's no auto-repartition (re-hashing would break partition-pinning and per-key ordering); repartition on the source first.

CAVEAT, not a block:

• max.message.bytes above the 1 MiB target cap. Config isn't data — your actual records may all be within the limit. A record that is over the cap doesn't fail assessment; it blocks and reports at replicate time instead (see Migrate below).

Not an assess blocker at all:

• replication.factor > 1. This is a constraint on how you migrate, not a fitness verdict — KubeMQ topics run at RF=1 (durability comes from the cluster itself, not per-topic replica count), so a replicated source topic migrates as an RF=1 target.

Active consumer groups on the source are tagged "requires exact offset translation at cutover" — stop their consumers before you run kmq migrate cutover.

The verdicts above are the same rubric rendered by the fitness matrix (T1–T4) — kmq assess kafka just maps it onto your actual topics, configs, and consumer groups.

Migrate — kmq migrate

kmq migrate mirrors the shape of a real migration, one command per phase:

The source is only ever read. Every phase dials the source with a read-only client — none of them produce, commit an offset, join a consumer group, or auto-create a topic there. All writes (topic creation, records, seeded offsets) land on the KubeMQ target only.

The assess phase is covered in detail above — see Assess fit — kmq assess kafka.

Replicate — copy the history

kmq migrate replicate \
  --bootstrap source-broker:9092 [source auth flags] \
  --target-bootstrap kubemq-broker:9092 [target auth flags] \
  --state ./migration.state

Copies every source topic-partition into KubeMQ byte-for-byte — key, value, headers, and the original CreateTime — partition-pinned (a record's source partition is preserved exactly; no key re-hash). It produces with acks=all and, in the same pass, records an exact per-record source-offset → target-offset map to --state.

• Resumable and crash-safe. Re-running with the same --state resumes from the last durably-copied offset per partition; already-acked records aren't re-copied.
• Oversized records block — they never skip. A source record larger than the target's message-size cap halts replication at that offset with a report (topic/partition/offset/size). The partition watermark doesn't advance past it, so nothing is silently dropped. Resolve the record (or raise the target's max-message-bytes, if appropriate), then re-run.
• Restrict scope with --topic (repeatable); the default is every non-internal source topic.

Translate — preview the offset map (no writes)

kmq migrate translate --bootstrap source-broker:9092 --target-bootstrap kubemq-broker:9092 --state ./migration.state

For each source consumer group, shows the target resume offset each committed offset maps to, and flags any group that can't be cut over cleanly. This is a read-only preview of exactly what cutover would seed — the second half of the staged dry-run described in the Overview.

Cutover — seed the offsets

# stop the source consumers first, then:
kmq migrate cutover --bootstrap source-broker:9092 --target-bootstrap kubemq-broker:9092 --state ./migration.state

Reads each source group's committed offsets, translates them through the offset map, and seeds them on the KubeMQ target via an empty-group offset commit — before any consumer joins. Point your consumers at KubeMQ and they resume at the correct position.

• Fail-closed completeness — zero tail loss on acked/committed offsets. A group is seeded only once replication has durably reached at least that group's committed offset on every partition. If a group consumed past what's been replicated, cutover refuses it with a clear reason, rather than seeding a position that would silently skip the un-replicated tail. Replicate further, then re-run.
• Offset-based only. Cutover resumes by offset, never by timestamp — a target timestamp-seek would resolve KubeMQ's own ingestion time, not the preserved source CreateTime, and would mis-position a consumer.
• Idempotent. Re-running overwrites the seeded offsets safely. --dry-run translates and checks completeness without writing; --force seeds even if the source group is still active (stop its consumers first — an active group is refused by default).

Per-source auth

The migration mechanics are identical across sources — only the auth flags differ, both for reading the source (assess/replicate/translate/cutover) and for the application's eventual switch to KubeMQ.

kmq migrate assess \
  --bootstrap b-1.your-cluster.kafka.us-east-1.amazonaws.com:9098 \
  --sasl-mechanism aws-msk-iam \
  --aws-access-key <access-key> \
  --aws-secret-key <secret-key> \
  [--aws-session-token <token>]

kmq migrate assess \
  --bootstrap pkc-xxxxx.us-east-1.aws.confluent.cloud:9092 \
  --tls \
  --sasl-mechanism plain \
  --sasl-username <api-key> \
  --sasl-password <api-secret>

kmq migrate assess \
  --bootstrap kafka.internal:9093 \
  --tls --tls-ca /path/to/ca.pem \
  --sasl-mechanism scram-sha-256 \
  --sasl-username <user> \
  --sasl-password <pass>

Large histories — MirrorMaker 2 hybrid

kmq migrate is the offset-exact vehicle — tuned for correct consumer resume, not for saturating a multi-terabyte historical backfill from a single process. For a very large history, use a hybrid:

• MirrorMaker 2 for the bulk data copy. MM2 replicates records into KubeMQ byte-perfectly, and its distributed workers scale the throughput. Use the shippable config: pin every *.replication.factor=1 (KubeMQ requires RF=1) and set offset-syncs.topic.location=target (the default writes that topic to your source cluster instead).
• kmq migrate cutover for the offsets. Don't rely on MM2 for consumer-offset translation — its own checkpoint translation does not produce a correct zero-loss resume position. Use MM2 for the data, then seed offsets with kmq migrate against the same target.

MM2 (data) + kmq migrate (offsets) gives you MM2's throughput and kmq migrate's exact consumer resume.

Rollback

Migration is non-destructive to the source — every phase only reads it. Rollback is:

1. Point bootstrap.servers back to the original source cluster.
2. Resume the source consumers.

Because the source's data and committed offsets are untouched throughout, it remains a complete, consistent fallback until you decommission it. The KubeMQ target can be discarded and the migration re-run from scratch.

OAUTHBEARER onboarding

If your Kafka clients already authenticate with OAUTHBEARER against an OIDC identity provider, KubeMQ's Kafka connector accepts the same mechanism natively — there's no separate credential model to bolt on for the migrated workload. See the Kafka OAUTHBEARER settings for the six-field OAuthBearer config block and its environment variables.

Verify the migration

Use a standard Kafka client pointed at the KubeMQ target to confirm the cutover worked.

1. Point a client at the KubeMQ target. Repoint bootstrap.servers (or kcat -b) to the KubeMQ Kafka listener you migrated into.

2. Produce a record to a migrated topic and confirm it's accepted:

   kcat -b kubemq-broker:9092 -t orders -P <<< "smoke-test-record"

3. Consume it back from the same topic:

   kcat -b kubemq-broker:9092 -t orders -C -c 1

4. Check a consumer-group offset. Resume a migrated group and confirm it picks up at the translated offset — not 0:

   kafka-consumer-groups.sh --bootstrap-server kubemq-broker:9092 --describe --group my-group

   The committed offset shown should match kmq migrate translate's preview for that group, and the group should resume without reprocessing already-consumed records.

See Also