# Licensing Troubleshooting (/licensing/troubleshooting)



Every licensing refusal, stop or warning the server logs ends with a link of the form
`https://docs.kubemq.io/licensing/troubleshooting#<anchor>`. Find the anchor below.

Two things to know before you start:

* The **wording** of each log line quoted below is illustrative; the **anchor** at the
  end of the line is fixed and is what to search for.

* A licensing exit is always **exit code 3**. Exit 1 is a durability fault and exit 0 a
  clean shutdown.

* `GET /api/v1/license` on the management port shows the live state — lease expiry,
  last refresh outcome, grace countdown, revocation count, clock skew and the exact
  warnings currently active. Most sections below tell you which field to read.

```bash title="Terminal"
curl -s http://localhost:8080/api/v1/license | jq
```

## Refusals at boot [#refusals-at-boot]

### No license input [#no-license-input]

**Log line:** `license: no license input — set KUBEMQ_LICENSE_KEY or KUBEMQ_LICENSE_FILE … #no-license-input`

**Cause.** None of `KUBEMQ_LICENSE_KEY`, `KUBEMQ_LICENSE_FILE` or `KUBEMQ_LICENSE_DATA`
is set. There is no unlicensed mode.

**Fix.** Set one of them. New here? Start a [trial](/licensing/trial). On Kubernetes
with the operator, set one of the four `spec.license*` fields
([Kubernetes](/licensing/kubernetes#crd-fields)). Older variable names and the
config-file license block are no longer read — see
[Coming from legacy v2](/deploy/coming-from-legacy-v2).

### Invalid signature [#invalid-signature]

**Log line:** `license: token rejected: <reason> … #invalid-signature`

**Cause.** The token failed verification: bad signature, unknown signing key id, wrong
algorithm, missing expiry, a forbidden claim, a malformed file header, or a cache file
(`<store>/license/lease.jws`) that was edited. The reason names which.

**Fix.**

* A file: make sure it is the complete, unmodified text between
  `-----BEGIN KUBEMQ LICENSE-----` and `-----END KUBEMQ LICENSE-----`; check that a
  templating step did not re-wrap or re-encode it. A legacy `-----BEGIN LICENSE KEY-----`
  file is rejected — request a new file.
* `unknown kid`: the server is older than the key that signed the token. Upgrade the
  server, or ask for a token signed with a key your version trusts (see
  [#untrusted-kid](#untrusted-kid)).
* A tampered cache: delete `<store>/license/lease.jws` and restart; the server
  re-activates with the key.

### Fingerprint mismatch [#fingerprint-mismatch]

**Log line:** `license: token is bound to fingerprint <a>, this installation is <b> … #fingerprint-mismatch`

**Cause.** An offline file's fingerprint list does not include this installation, or
the licensing service returned a lease bound to a different installation. Typical
triggers: a store volume moved between machines; on Kubernetes, the server could not
read the `kube-system` namespace and fell back to a random id
([#missing-kube-system-permission](#missing-kube-system-permission)); a container
restarted without its volume, which generated a new id.

For an **online** license a *cached* lease bound to another fingerprint (a lease cache
copied from another host, a container replaced without its volume) is **not** a
refusal: the server logs this anchor as a boot warning and activates again with the
key under its own fingerprint. The refusal happens only for an offline file, or when
the lease the service itself just issued does not match.

**Fix.** For a file, establish the actual fingerprint (on Kubernetes,
`kubectl get namespace kube-system -o jsonpath='{.metadata.uid}'`; elsewhere
`<store>/license/instance.id`, or `GET /api/v1/license` on a server running with an
online key) and request a reissued file that includes it. On Kubernetes, fix the RBAC
first so the fingerprint is the cluster's. For the online warning nothing is needed;
the re-activation counts as a new installation against your cap, so stop copying
store volumes between hosts.

### Offline file expired [#offline-expired]

**Log line:** `license: offline license expired at <time> … #offline-expired`

**Cause.** The file's `exp` has passed (5-minute leeway). At boot this is a refusal;
on a running server it is a drain and exit. Offline files have no grace period.

**Fix.** Request a renewed file, replace it, restart. See
[Renewal](/licensing/air-gap#renewal). Plan reissues from `lease_expires_at`.

### First boot, bad request [#first-boot-bad-request]

**Log line:** `license: activation refused (bad_request) 3 times … #first-boot-bad-request`

**Cause.** The licensing service rejected the activation body itself (`400 bad_request`)
three times. This is not about the key: the request shape the server sent was refused,
which points at a version mismatch between this server build and the licensing service.

**Fix.** Upgrade to the current official image. If the current image fails the same way,
report it to KubeMQ with the server version from the log line — nothing on your side
changes the outcome.

### First boot, bad key [#first-boot-bad-key]

**Log line:** `license: activation refused (invalid_key) 3 times … #first-boot-bad-key`

**Cause.** The licensing service answered `401 invalid_key` three times within a
minute. The same answer is given for a mistyped key, a key that was regenerated (so the
old one is dead), a disabled or revoked license, and a license past its commercial
expiry — the service does not say which.

**Fix.** Check the value actually reaching the container (`docker inspect`, or the
Secret on Kubernetes) against the most recent key you were sent. If the key was
regenerated, update every place it is stored. If it is correct and current, contact
KubeMQ — the license itself is disabled or expired.

### First boot, service unreachable [#first-boot-unreachable]

**Log line:** `license: could not reach https://license.kubemq.io for 30 minutes … #first-boot-unreachable`

**Cause.** This installation has no cached lease and activation kept failing with a
network-class error (DNS, timeout, connection refused, 5xx) for
`KUBEMQ_LICENSE_FIRST_BOOT_RETRY_MINUTES` (default 30). A first boot **must** reach the
service; only a running server with a cache can fail open.

**Fix.**

* Allow HTTPS egress to `license.kubemq.io`. Verify from inside
  the container network: `curl -sS https://license.kubemq.io/api/healthz`.
* If `KUBEMQ_LICENSE_ENDPOINT` is set, make sure it is correct and has no trailing
  slash.
* If egress cannot be allowed, request a
  [pre-activated file](/licensing/license-key#pre-activated-file-for-egress-restricted-networks).
* If the cache existed before, the store is not persistent —
  see [#store-not-persistent](#store-not-persistent).

### Both key and file set [#both-key-and-file]

**Log line:** `license: both KUBEMQ_LICENSE_KEY and KUBEMQ_LICENSE_FILE are set; using the file … #both-key-and-file`

**Cause.** A warning, not a refusal. Both inputs are present; the file wins and the key
is ignored, so the server runs offline: no refresh, no usage reports.

**Fix.** Remove the input you do not intend. If you meant to run online, unset the file
variables; if you meant offline, unset the key.

### Cluster over cap [#cluster-over-cap]

**Log line:** `license: cluster has <n> members, license allows <max_instances> … #cluster-over-cap`

**Cause.** At boot, the statically configured members — the next engine's peer list
(every initial member including this one), or the legacy route list plus this node —
outnumber the license's `max_instances`: a refusal. Or, on a live next-engine cluster
that joins by membership (the `Cluster.Replication.Join` setting), the voting members
have been over the cap for 30 consecutive one-minute readings: this node drains.

**Fix.** Reduce replicas to the cap, or ask for a license with a higher cap. On
Kubernetes with the operator the clamp on first create prevents this whenever the
operator knows the cap (an offline file, or a cached lease); a later `spec.replicas`
increase past the cap is blocked with the `ReplicasOverLicenseCap` condition. Check
`max_instances` in `GET /api/v1/license`.

## Running server: stops and warnings [#running-server-stops-and-warnings]

### Lease and grace exhausted [#lease-and-grace-exhausted]

**Log line:** `license: lease expired <time> and grace of <n> days exhausted; draining … #lease-and-grace-exhausted`

**Cause.** The server has not obtained a new lease for the whole of the lease (7 days;
24 h for trials) plus `grace_days`. Either the service was unreachable for that long,
or it stopped issuing leases: the license is disabled, past its commercial expiry, or
suspended for 30 days of silence
([#lease-refused](#lease-refused), [#silent-usage](#silent-usage)). The server drains
for `KUBEMQ_LICENSE_DRAIN_SECONDS` and exits with code 3.

**Fix.** Look at `last_refresh.outcome` in `GET /api/v1/license` *before* the stop, or
in the log history: `unreachable` means a network problem — restore egress and restart;
`unsigned_4xx` with `lease_refused` means the license needs renewing — contact KubeMQ,
then restart. A trial that ended converts through the reminder email.

### Grace countdown [#grace-countdown]

**Log line:** `license: lease expired <time>; running on grace, <n> days left … #grace-countdown`

**Cause.** A warning every minute. The lease's `exp` has passed and the service is
still unreachable (or refusing), but you are inside `grace_days`. The server runs
normally.

**Fix.** Treat it as an outage with a deadline: `run_until` in `GET /api/v1/license` is
the moment it stops. Restore connectivity; a single successful refresh ends the
countdown and issues a fresh 7-day lease.

### Revoked [#revoked]

**Log line:** `license: revocation confirmed (3 assertions): <reason>; draining … #revoked`

**Cause.** The licensing service has answered three refreshes, at least 10 minutes
apart, with a signed revocation for this license — the result of a deliberate "revoke
now" or "enforce cap" action by KubeMQ. The server drains and exits with code 3, and
will refuse to start again with the same license.

**Fix.** Contact KubeMQ. `revocation.reason` in `GET /api/v1/license` carries the
reason given. The three confirmations are persisted in `<store>/license/revocation.json`,
so the server refuses to boot with that license until KubeMQ reinstates it — a
reinstated license is reissued with a newer `token_version`. To apply it, delete both
`<store>/license/lease.jws` and `<store>/license/revocation.json`, update the key if it
changed, and restart; the server activates again and the new lease clears the record.

### Revocation seen [#revocation-seen]

**Log line:** `license: revocation assertion seen (<n> of 3): <reason> … #revocation-seen`

**Cause.** A warning. One or two signed revocations have been received; the server now
re-polls every 10 minutes and will drain after the third. A stray or replayed response
cannot cause this — only a validly signed, fresh assertion counts, and an assertion is
counted only when both its issue time and its arrival are at least 10 minutes after the
previously counted one. The count is persisted, so a restart with one or two on record
resumes the 10-minute polling with the same count.

**Fix.** Contact KubeMQ immediately if this is unexpected. `revocation.seen_count` and
`revocation.first_seen` in `GET /api/v1/license` show the timeline.

### Clock rollback [#clock-rollback]

**Log line:** `license: wall clock <now> is behind the persisted high-water mark <hwm> … #clock-rollback`

**Cause.** The system clock is more than 10 minutes *earlier* than the latest local
time this server has ever seen (persisted in `<store>/license/clock.hwm`). This blocks
a clock set back to extend a lease. What happens next depends on the mode:

* **Online.** The server asks the licensing service to confirm: at boot it attempts a
  refresh and refuses to start if that refresh does not return a signed lease; on a
  running server it requests one refresh and waits for its outcome — a signed lease
  clears the condition, a `429 rate_limited` answer defers the decision to the next
  scheduled refresh, and any other outcome (unreachable, refused) stops the server. A
  successful refresh resets the mark to the **local** clock — the signed lease is what
  validates local time; the service's `server_time` is used only for the skew figure.
* **Offline.** There is no service to ask. The server refuses to start, or stops if it
  is running.

**Fix.** Fix the clock (NTP) first; that is almost always the cause. If the clock is
right and the mark is wrong (for example a store restored from a backup taken on a
machine with a bad clock), stop the server, delete `<store>/license/clock.hwm`, and
start it again; the server re-persists the mark from its local clock.

### Clock skew [#clock-skew]

**Log line:** `license: local clock differs from the licensing service by <n> s … #clock-skew`

**Cause.** The local time and the service's time differ by more than 5 minutes. The
service's time is taken only from a response that carried a signed lease: the lease's
signed `iat`, refined by the unsigned `server_time` field when that agrees with the
`iat` within 5 minutes (a `server_time` further off is ignored, and logged under this
anchor). A warning, and `clock.skew_seconds` in `GET /api/v1/license` shows the value.
Skew above 24 hours becomes a stop once the lease has expired. The same anchor is used
when a usage report is answered `413` or `422` — the row is dropped as terminal,
typically because its timestamps are out of range.

**Fix.** Synchronize the clock with NTP. Skew never extends a lease: expiry is
enforced from lease `exp` + grace regardless.

### Over cap [#over-cap]

**Log line:** `license: <n> active installations, license allows <max_instances> … #over-cap`

**Cause.** A warning every minute. The lease carries `over_cap: true` — more
installations (distinct fingerprints, or concurrent server processes) have reported
than the license allows. Nothing is enforced automatically; the situation is visible to
KubeMQ, who may contact you or, after notice, revoke the newest installations.

**Fix.** Stop the extra installations, or ask for a higher cap. Common accidental
causes: containers started without a store volume (each restart is a new fingerprint),
and Kubernetes pods without the `kube-system` permission (each pod counts separately).

### Silent usage [#silent-usage]

**Log line:** `license: no usage report accepted for 72 h … #silent-usage`

**Cause.** A warning every minute. Leases are still being refreshed but the service
has not accepted a usage report from this license for 72 hours — usually because
`POST /v1/usage/report` is blocked while `/v1/license/refresh` is not, or every report
is rejected. While silent, leases are shortened to 24 hours; after 30 days of silence
the service stops issuing leases (`503 silence_suspended`) and the server runs out its
lease and grace.

**Fix.** Check `usage.last_outcome` and `usage.last_status` in `GET /api/v1/license`.
`error` means the request is not getting through — allow the same egress you allow for
refresh. `rejected` with a 422 means a clock problem ([#clock-skew](#clock-skew)). See
[Usage reporting](/licensing/usage-reporting) for what is sent.

### Store not persistent [#store-not-persistent]

**Log line:** `license: store path <path> is not a mounted volume; the license cache will not survive a restart … #store-not-persistent`

**Cause.** A boot warning. The store directory is on the same filesystem as `/`, which
inside a container means it dies with the container. The lease cache, the instance id
and the usage cursor are all under it.

**Fix.** Mount a volume at the store path (`/kubemq/store` in the official image) —
`-v kubemq-data:/kubemq/store` on Docker, a `volumeClaimTemplate` on Kubernetes. See
[The lease cache](/licensing/license-key#the-lease-cache). A throwaway run can ignore
the warning.

### Missing kube-system permission [#missing-kube-system-permission]

**Log line:** `license: could not read namespace kube-system (<error>); using persisted instance id … #missing-kube-system-permission`

**Cause.** A boot warning on Kubernetes. The server's ServiceAccount lacks `get` on
`namespaces` for `kube-system` (or the API server was unreachable), so the fingerprint
is a per-pod persisted id instead of the cluster's namespace UID.
`fingerprint_source` reads `persisted`.

**Fix.** Bind the `kubemq-cluster-next-license` ClusterRole to the pod's ServiceAccount
— the chart does this with the `kubemq-cluster-next-<namespace>-license-crb` binding;
the manifest is in
[Kubernetes without the operator](/licensing/kubernetes#rbac-for-the-fingerprint).
Then restart the pods. Until then every pod is a separate installation, and an offline
file bound to the cluster UID will not match.

### Untrusted signing key [#untrusted-kid]

**Log line:** `license: the licensing service has no signing key this server trusts (422 untrusted_kid) … #untrusted-kid`

**Cause.** The server's built-in set of trusted public keys is older than the service's
current signing key. The service refuses to sign with a key the server would reject.
On a **first boot** (no cached lease) this is an immediate refusal — there is no lease
to fall back on. A **running** server keeps its cached lease and runs until lease +
grace.

**Fix.** Upgrade the server to a release that ships the new key. Key rotations are
announced in the release notes at least two releases ahead.

### Lease refused [#lease-refused]

**Log line:** `license: refresh answered HTTP 401 lease_refused (unsigned); keeping the cached lease until <run_until> … #lease-refused`

**Cause.** The cached lease is valid but the refresh did not produce a new one. This
anchor covers every unsigned answer to a refresh:

* `401 lease_refused` — the service will not issue a lease: the license has been
  **disabled**, or its **commercial expiry** has passed.
* any other unsigned `4xx` (`last_refresh.outcome: unsigned_4xx`), except
  `422 untrusted_kid` ([#untrusted-kid](#untrusted-kid)).
* a `5xx` (`unsigned_5xx`) — a service-side error, except `503 silence_suspended`
  ([#silent-usage](#silent-usage)).
* a `200` whose body is not a signed lease (`malformed_200`) — almost always a proxy or
  captive portal answering in place of `license.kubemq.io`.

None of these is signed, so none stops the server immediately — it runs on the cached
lease, retries with backoff, and drains at lease `exp` + grace
([#lease-and-grace-exhausted](#lease-and-grace-exhausted)).

**Fix.** Read `last_refresh.outcome` and `last_refresh.status` in `GET /api/v1/license`.
For `401`, compare `license_expires_at` with today: if it is in the past, renew with
KubeMQ — the next refresh after renewal picks up the new expiry with no restart; if it
is in the future, the license was disabled — contact KubeMQ. For `malformed_200`, check
what sits between the server and the endpoint. For a `5xx`, wait; the server keeps
retrying.

## Quick reference [#quick-reference]

| Anchor                            | Refusal or warning                                                    | First thing to check                           |
| --------------------------------- | --------------------------------------------------------------------- | ---------------------------------------------- |
| `#no-license-input`               | Refusal at boot                                                       | The environment reaching the process           |
| `#invalid-signature`              | Refusal at boot                                                       | File integrity; `lease.jws` untouched          |
| `#fingerprint-mismatch`           | Refusal at boot (file); warning + re-activation (cached online lease) | `fingerprint` vs the token; RBAC on Kubernetes |
| `#offline-expired`                | Refusal / stop                                                        | `lease_expires_at`                             |
| `#first-boot-bad-request`         | Refusal at boot                                                       | Server version vs the licensing service        |
| `#first-boot-bad-key`             | Refusal at boot                                                       | The key value                                  |
| `#first-boot-unreachable`         | Refusal at boot                                                       | Egress to `license.kubemq.io`                  |
| `#both-key-and-file`              | Warning                                                               | Which input you meant                          |
| `#cluster-over-cap`               | Refusal / stop                                                        | `max_instances` vs replicas                    |
| `#lease-and-grace-exhausted`      | Stop                                                                  | `last_refresh.outcome`                         |
| `#grace-countdown`                | Warning                                                               | `run_until`; connectivity                      |
| `#revoked`                        | Stop                                                                  | `revocation.reason`                            |
| `#revocation-seen`                | Warning                                                               | `revocation.seen_count`                        |
| `#clock-rollback`                 | Refusal / stop                                                        | NTP; `clock.hwm`                               |
| `#clock-skew`                     | Warning                                                               | NTP; `clock.skew_seconds`                      |
| `#over-cap`                       | Warning                                                               | Installation count; store volumes              |
| `#silent-usage`                   | Warning                                                               | `usage.last_outcome`; egress for usage         |
| `#store-not-persistent`           | Warning                                                               | The volume mount                               |
| `#missing-kube-system-permission` | Warning                                                               | The ClusterRole binding                        |
| `#untrusted-kid`                  | Refusal at first boot; warning on a running server                    | Server version                                 |
| `#lease-refused`                  | Warning                                                               | `last_refresh.outcome`; `license_expires_at`   |