# Migrate from Kafka (/connectors/kafka/how-to/migrate-from-kafka)
**Drop-in level: endpoint-only.** KubeMQ speaks the native Kafka wire protocol — real
`librdkafka`/`kcat`/Java clients connect unchanged; you repoint `bootstrap.servers`, with no
client-library swap or code change. Moving an *existing* Apache Kafka, Amazon MSK, or Confluent
cluster onto KubeMQ is more than pointing at a new endpoint if you want consumers to resume
without reprocessing history — that historical topic-data and offset move is what the separate
`kmq migrate` tool below handles. This guide covers the read-only fitness check, the four-phase `kmq migrate` tool,
per-source auth, the large-history MirrorMaker 2 hybrid, rollback, and a post-cutover smoke
test.
## Overview [#overview]
Migrating an existing Kafka workload onto KubeMQ has two stages: **assess**, then
**migrate**. `kmq assess kafka` scans your source cluster read-only and reports, per topic,
whether it's a straight repoint or needs a workaround. `kmq migrate` then does the actual
work — copying topic history and consumer-group offsets to KubeMQ in four phases (assess →
replicate → translate → cutover) — so consumers resume where they left off instead of
reprocessing everything from scratch.
**Know `kmq migrate`'s scope before you rely on it.** Its engine — byte-fidelity
replication, exact per-record offset translation, the cutover-completeness gate, and
oversized-record block-and-report — is proven on a real 3-node `next` cluster: a SIGKILL
mid-replication loses nothing, and a kill during cutover never double-seeds an offset.
Cluster-level consumer-resume is proven with a real `franz-go` client; Java/kcat/librdkafka
consumer-resume is proven **single-node** only — the full multi-client (Java / librdkafka /
franz-go) cluster consumer-resume run is what remains. The MSK and Confluent auth procedures
below are docs-grounded: validated against a local Apache Kafka, with cloud-specific details
taken from each vendor's own documentation.
**Do a staged dry-run before any production cutover.** Run `kmq migrate assess` and
`kmq migrate translate` against a copy of your data first and confirm the verdicts and the
offset-map preview look right — before you point a single production consumer at the
KubeMQ target.
## Storage engine (zero-config) [#storage-engine-zero-config]
Kafka on KubeMQ requires the **`next`** storage engine — but you don't set it manually. On a
**fresh** deployment, enabling the Kafka connector with the engine left unset **auto-selects
`next`** and logs a `NOTICE`; there is no separate "set `store.engine=next` first" step.
| Case | Behavior |
| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Fresh deployment**, Kafka enabled, engine unset | Auto-selects `next`. Zero-config — nothing to set before you enable Kafka. |
| **Existing `legacy` cluster**, Kafka enabled | Cannot be retrofitted. The server fails closed at boot with a config error naming the conflicting store directory — there is no in-place engine migration between `legacy` and `next`. Stand up a new cluster on `next` instead. |
| **Explicit `STORE_ENGINE=legacy`**, Kafka enabled | Rejected at boot — pinning `legacy` alongside Kafka is a configuration error. |
Pinning `STORE_ENGINE=next` (or `store.engine: next` in `config.yaml`) always works and
always wins — it skips the auto-select probe entirely, which is the predictable choice for
IaC/GitOps. See [Storage Engines](/configure/reference/storage-engines#zero-config-engine-selection)
for the full engine model, mode isolation, and durability guarantees.
## Assess fit — `kmq assess kafka` [#assess-fit--kmq-assess-kafka]
Before touching anything, run the read-only assessor against your source cluster:
```bash
kmq assess kafka --bootstrap your-broker:9092 [--tls --sasl-mechanism scram-sha-256 --sasl-username --sasl-password ]
```
`kmq assess kafka` never produces, commits an offset, or creates a topic — it's safe to run
against a production cluster. It reports, per topic, one of four verdicts, plus a single
overall `migratable` verdict:
| Verdict | Meaning |
| ----------- | ------------------------------------------------------------------------------------------------------------------ |
| **READY** | No obstruction found — a straight repoint. |
| **CAVEAT** | Migratable, with a documented constraint (see below). |
| **UNKNOWN** | The topic's config couldn't be read (for example, a restricted cluster). **Fail-safe** — never assumed migratable. |
| **BLOCKED** | A hard blocker (see below) — this topic won't migrate through `kmq migrate`. |
**Hard BLOCKED:**
* **Compacted topics** (`cleanup.policy=compact`) — compaction leaves non-contiguous offsets
(compaction holes) that the exact offset-translation path can't follow. Move these via bulk
data copy (the [MirrorMaker 2 hybrid](#large-histories--mirrormaker-2-hybrid) below) and
rebuild consumer state, or keep them on the source.
* **More than 256 partitions on a topic** — KubeMQ's per-topic partition cap. There's no
auto-repartition (re-hashing would break partition-pinning and per-key ordering); repartition
on the source first.
**CAVEAT, not a block:**
* **`max.message.bytes` above the 1 MiB target cap.** Config isn't data — your actual records
may all be within the limit. A record that *is* over the cap doesn't fail assessment; it
blocks and reports at replicate time instead (see [Migrate](#migrate--kmq-migrate)
below).
**Not an assess blocker at all:**
* **`replication.factor > 1`.** This is a constraint on *how* you migrate, not a fitness
verdict — KubeMQ topics run at RF=1 (durability comes from the cluster itself, not per-topic
replica count), so a replicated source topic migrates as an RF=1 target.
Active consumer groups on the source are tagged "requires exact offset translation at
cutover" — stop their consumers before you run `kmq migrate cutover`.
The verdicts above are the same rubric rendered by the
[fitness matrix](/connectors/kafka/reference/fitness-matrix) (T1–T4) — `kmq assess kafka`
just maps it onto your actual topics, configs, and consumer groups.
## Migrate — `kmq migrate` [#migrate--kmq-migrate]
`kmq migrate` mirrors the shape of a real migration, one command per phase:
| Phase | Command | Touches the source | Touches the target | Reversible |
| ------------- | ----------------------- | ------------------ | ----------------------- | ------------------ |
| **assess** | `kmq migrate assess` | read-only | — | Yes — read-only |
| **replicate** | `kmq migrate replicate` | read-only | writes topics + records | Yes — target only |
| **translate** | `kmq migrate translate` | read-only | — | Yes — preview only |
| **cutover** | `kmq migrate cutover` | read-only | writes group offsets | Yes — re-runnable |
**The source is only ever read.** Every phase dials the source with a read-only client — none
of them produce, commit an offset, join a consumer group, or auto-create a topic there. All
writes (topic creation, records, seeded offsets) land on the KubeMQ **target** only.
The **assess** phase is covered in detail above — see
[Assess fit — `kmq assess kafka`](#assess-fit--kmq-assess-kafka).
### Replicate — copy the history [#replicate--copy-the-history]
```bash
kmq migrate replicate \
--bootstrap source-broker:9092 [source auth flags] \
--target-bootstrap kubemq-broker:9092 [target auth flags] \
--state ./migration.state
```
Copies every source topic-partition into KubeMQ **byte-for-byte** — key, value, headers, and
the original `CreateTime` — **partition-pinned** (a record's source partition is preserved
exactly; no key re-hash). It produces with `acks=all` and, in the same pass, records an exact
per-record source-offset → target-offset map to `--state`.
* **Resumable and crash-safe.** Re-running with the same `--state` resumes from the last
durably-copied offset per partition; already-acked records aren't re-copied.
* **Oversized records block — they never skip.** A source record larger than the target's
message-size cap halts replication at that offset with a report (topic/partition/offset/size).
The partition watermark doesn't advance past it, so nothing is silently dropped. Resolve the
record (or raise the target's max-message-bytes, if appropriate), then re-run.
* Restrict scope with `--topic` (repeatable); the default is every non-internal source topic.
### Translate — preview the offset map (no writes) [#translate--preview-the-offset-map-no-writes]
```bash
kmq migrate translate --bootstrap source-broker:9092 --target-bootstrap kubemq-broker:9092 --state ./migration.state
```
For each source consumer group, shows the target resume offset each committed offset maps to,
and flags any group that can't be cut over cleanly. This is a read-only preview of exactly
what `cutover` would seed — the second half of the staged dry-run described in the
[Overview](#overview).
### Cutover — seed the offsets [#cutover--seed-the-offsets]
```bash
# stop the source consumers first, then:
kmq migrate cutover --bootstrap source-broker:9092 --target-bootstrap kubemq-broker:9092 --state ./migration.state
```
Reads each source group's committed offsets, translates them through the offset map, and
seeds them on the KubeMQ target via an empty-group offset commit — before any consumer joins.
Point your consumers at KubeMQ and they resume at the correct position.
* **Fail-closed completeness — zero tail loss on acked/committed offsets.** A group is seeded
only once replication has durably reached at least that group's committed offset on every
partition. If a group consumed past what's been replicated, cutover **refuses** it with a
clear reason, rather than seeding a position that would silently skip the un-replicated
tail. Replicate further, then re-run.
* **Offset-based only.** Cutover resumes by offset, never by timestamp — a target
timestamp-seek would resolve KubeMQ's own ingestion time, not the preserved source
`CreateTime`, and would mis-position a consumer.
* **Idempotent.** Re-running overwrites the seeded offsets safely. `--dry-run` translates and
checks completeness without writing; `--force` seeds even if the source group is still
active (stop its consumers first — an active group is refused by default).
## Per-source auth [#per-source-auth]
The migration mechanics are identical across sources — only the **auth flags** differ, both
for reading the source (`assess`/`replicate`/`translate`/`cutover`) and for the application's
eventual switch to KubeMQ.
| Source | Reading the source | After cutover, on KubeMQ |
| ----------------------------- | ---------------------------------------------------------- | ---------------------------------------------------- |
| **Amazon MSK** | MSK IAM — `--sasl-mechanism aws-msk-iam` with static keys | SASL/SCRAM or mTLS |
| **Confluent Cloud** | SASL/PLAIN over TLS — API key/secret as username/password | SASL/PLAIN-over-TLS (rotate the API-key credentials) |
| **Self-managed Apache Kafka** | PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512 (+ TLS as required) | SASL/SCRAM, PLAIN, mTLS, or OAUTHBEARER |
MSK IAM is `--sasl-mechanism aws-msk-iam` with **static keys** — the AWS default credential
chain (profile / SSO / IMDS) is not used; supply the keys explicitly:
```bash
kmq migrate assess \
--bootstrap b-1.your-cluster.kafka.us-east-1.amazonaws.com:9098 \
--sasl-mechanism aws-msk-iam \
--aws-access-key \
--aws-secret-key \
[--aws-session-token ]
```
The application-side switch is config-only: IAM/SigV4 on the source becomes SASL/SCRAM or
mTLS on KubeMQ. Your code is unchanged beyond the `bootstrap.servers` value you were already
changing at migration.
Confluent Cloud is SASL/PLAIN over TLS — **not** SCRAM — with the API key/secret as the
username/password:
```bash
kmq migrate assess \
--bootstrap pkc-xxxxx.us-east-1.aws.confluent.cloud:9092 \
--tls \
--sasl-mechanism plain \
--sasl-username \
--sasl-password
```
After cutover, rotate the Confluent API-key credentials to KubeMQ SASL/PLAIN-over-TLS. If you
run Schema Registry, it keeps working unchanged against KubeMQ.
Self-managed Apache Kafka uses whichever SASL mechanism your cluster already runs:
```bash
kmq migrate assess \
--bootstrap kafka.internal:9093 \
--tls --tls-ca /path/to/ca.pem \
--sasl-mechanism scram-sha-256 \
--sasl-username \
--sasl-password
```
**Kerberos / GSSAPI caveat.** The bridge can *consume* a GSSAPI-secured source where the
underlying client library supports it — but **KubeMQ itself doesn't serve Kerberos**. The
migrated workload authenticates to KubeMQ with SASL/SCRAM, PLAIN, mTLS, or OAUTHBEARER
instead.
## Large histories — MirrorMaker 2 hybrid [#large-histories--mirrormaker-2-hybrid]
`kmq migrate` is the offset-exact vehicle — tuned for correct consumer resume, not for
saturating a multi-terabyte historical backfill from a single process. For a very large
history, use a hybrid:
* **MirrorMaker 2** for the bulk **data** copy. MM2 replicates records into KubeMQ
byte-perfectly, and its distributed workers scale the throughput. Use the shippable
config: pin every `*.replication.factor=1` (KubeMQ requires RF=1) and set
`offset-syncs.topic.location=target` (the default writes that topic to your *source*
cluster instead).
* **`kmq migrate cutover`** for the **offsets**. Don't rely on MM2 for consumer-offset
translation — its own checkpoint translation does not produce a correct zero-loss resume
position. Use MM2 for the data, then seed offsets with `kmq migrate` against the same
target.
MM2 (data) + `kmq migrate` (offsets) gives you MM2's throughput **and** `kmq migrate`'s exact
consumer resume.
## Rollback [#rollback]
Migration is **non-destructive to the source** — every phase only reads it. Rollback is:
1. Point `bootstrap.servers` back to the original source cluster.
2. Resume the source consumers.
Because the source's data and committed offsets are untouched throughout, it remains a
complete, consistent fallback until you decommission it. The KubeMQ target can be discarded
and the migration re-run from scratch.
## OAUTHBEARER onboarding [#oauthbearer-onboarding]
If your Kafka clients already authenticate with OAUTHBEARER against an OIDC identity
provider, KubeMQ's Kafka connector accepts the same mechanism natively — there's no separate
credential model to bolt on for the migrated workload. See
[the Kafka OAUTHBEARER settings](/configure/reference/connectors#kafka) for the
six-field `OAuthBearer` config block and its environment variables.
## Verify the migration [#verify-the-migration]
Use a standard Kafka client pointed at the KubeMQ target to confirm the cutover worked.
1. **Point a client at the KubeMQ target.** Repoint `bootstrap.servers` (or `kcat -b`) to the
KubeMQ Kafka listener you migrated into.
2. **Produce a record** to a migrated topic and confirm it's accepted:
```bash
kcat -b kubemq-broker:9092 -t orders -P <<< "smoke-test-record"
```
3. **Consume it back** from the same topic:
```bash
kcat -b kubemq-broker:9092 -t orders -C -c 1
```
4. **Check a consumer-group offset.** Resume a migrated group and confirm it picks up at the
translated offset — not `0`:
```bash
kafka-consumer-groups.sh --bootstrap-server kubemq-broker:9092 --describe --group my-group
```
The committed offset shown should match `kmq migrate translate`'s preview for that group,
and the group should resume without reprocessing already-consumed records.
## See Also [#see-also]