Licensing Troubleshooting

Every licensing refusal, stop or warning the server logs ends with a link of the form https://docs.kubemq.io/licensing/troubleshooting#<anchor>. Find the anchor below.

Two things to know before you start:

• The wording of each log line quoted below is illustrative; the anchor at the end of the line is fixed and is what to search for.

• A licensing exit is always exit code 3. Exit 1 is a durability fault and exit 0 a clean shutdown.

• GET /api/v1/license on the management port shows the live state — lease expiry, last refresh outcome, grace countdown, revocation count, clock skew and the exact warnings currently active. Most sections below tell you which field to read.

curl -s http://localhost:8080/api/v1/license | jq

Refusals at boot

No license input

Log line: license: no license input — set KUBEMQ_LICENSE_KEY or KUBEMQ_LICENSE_FILE … #no-license-input

Cause. None of KUBEMQ_LICENSE_KEY, KUBEMQ_LICENSE_FILE or KUBEMQ_LICENSE_DATA is set. There is no unlicensed mode.

Fix. Set one of them. New here? Start a trial. On Kubernetes with the operator, set one of the four spec.license* fields (Kubernetes). Older variable names and the config-file license block are no longer read — see Coming from legacy v2.

Invalid signature

Log line: license: token rejected: <reason> … #invalid-signature

Cause. The token failed verification: bad signature, unknown signing key id, wrong algorithm, missing expiry, a forbidden claim, a malformed file header, or a cache file (<store>/license/lease.jws) that was edited. The reason names which.

Fix.

• A file: make sure it is the complete, unmodified text between -----BEGIN KUBEMQ LICENSE----- and -----END KUBEMQ LICENSE-----; check that a templating step did not re-wrap or re-encode it. A legacy -----BEGIN LICENSE KEY----- file is rejected — request a new file.
• unknown kid: the server is older than the key that signed the token. Upgrade the server, or ask for a token signed with a key your version trusts (see #untrusted-kid).
• A tampered cache: delete <store>/license/lease.jws and restart; the server re-activates with the key.

Fingerprint mismatch

Log line: license: token is bound to fingerprint <a>, this installation is <b> … #fingerprint-mismatch

Cause. An offline file's fingerprint list does not include this installation, or the licensing service returned a lease bound to a different installation. Typical triggers: a store volume moved between machines; on Kubernetes, the server could not read the kube-system namespace and fell back to a random id (#missing-kube-system-permission); a container restarted without its volume, which generated a new id.

For an online license a cached lease bound to another fingerprint (a lease cache copied from another host, a container replaced without its volume) is not a refusal: the server logs this anchor as a boot warning and activates again with the key under its own fingerprint. The refusal happens only for an offline file, or when the lease the service itself just issued does not match.

Fix. For a file, establish the actual fingerprint (on Kubernetes, kubectl get namespace kube-system -o jsonpath='{.metadata.uid}'; elsewhere <store>/license/instance.id, or GET /api/v1/license on a server running with an online key) and request a reissued file that includes it. On Kubernetes, fix the RBAC first so the fingerprint is the cluster's. For the online warning nothing is needed; the re-activation counts as a new installation against your cap, so stop copying store volumes between hosts.

Offline file expired

Log line: license: offline license expired at <time> … #offline-expired

Cause. The file's exp has passed (5-minute leeway). At boot this is a refusal; on a running server it is a drain and exit. Offline files have no grace period.

Fix. Request a renewed file, replace it, restart. See Renewal. Plan reissues from lease_expires_at.

First boot, bad request

Log line: license: activation refused (bad_request) 3 times … #first-boot-bad-request

Cause. The licensing service rejected the activation body itself (400 bad_request) three times. This is not about the key: the request shape the server sent was refused, which points at a version mismatch between this server build and the licensing service.

Fix. Upgrade to the current official image. If the current image fails the same way, report it to KubeMQ with the server version from the log line — nothing on your side changes the outcome.

First boot, bad key

Log line: license: activation refused (invalid_key) 3 times … #first-boot-bad-key

Cause. The licensing service answered 401 invalid_key three times within a minute. The same answer is given for a mistyped key, a key that was regenerated (so the old one is dead), a disabled or revoked license, and a license past its commercial expiry — the service does not say which.

Fix. Check the value actually reaching the container (docker inspect, or the Secret on Kubernetes) against the most recent key you were sent. If the key was regenerated, update every place it is stored. If it is correct and current, contact KubeMQ — the license itself is disabled or expired.

First boot, service unreachable

Log line: license: could not reach https://license.kubemq.io for 30 minutes … #first-boot-unreachable

Cause. This installation has no cached lease and activation kept failing with a network-class error (DNS, timeout, connection refused, 5xx) for KUBEMQ_LICENSE_FIRST_BOOT_RETRY_MINUTES (default 30). A first boot must reach the service; only a running server with a cache can fail open.

Fix.

• Allow HTTPS egress to license.kubemq.io. Verify from inside the container network: curl -sS https://license.kubemq.io/api/healthz.
• If KUBEMQ_LICENSE_ENDPOINT is set, make sure it is correct and has no trailing slash.
• If egress cannot be allowed, request a pre-activated file.
• If the cache existed before, the store is not persistent — see #store-not-persistent.

Both key and file set

Log line: license: both KUBEMQ_LICENSE_KEY and KUBEMQ_LICENSE_FILE are set; using the file … #both-key-and-file

Cause. A warning, not a refusal. Both inputs are present; the file wins and the key is ignored, so the server runs offline: no refresh, no usage reports.

Fix. Remove the input you do not intend. If you meant to run online, unset the file variables; if you meant offline, unset the key.

Cluster over cap

Log line: license: cluster has <n> members, license allows <max_instances> … #cluster-over-cap

Cause. At boot, the statically configured members — the next engine's peer list (every initial member including this one), or the legacy route list plus this node — outnumber the license's max_instances: a refusal. Or, on a live next-engine cluster that joins by membership (the Cluster.Replication.Join setting), the voting members have been over the cap for 30 consecutive one-minute readings: this node drains.

Fix. Reduce replicas to the cap, or ask for a license with a higher cap. On Kubernetes with the operator the clamp on first create prevents this whenever the operator knows the cap (an offline file, or a cached lease); a later spec.replicas increase past the cap is blocked with the ReplicasOverLicenseCap condition. Check max_instances in GET /api/v1/license.

Running server: stops and warnings

Lease and grace exhausted

Log line: license: lease expired <time> and grace of <n> days exhausted; draining … #lease-and-grace-exhausted

Cause. The server has not obtained a new lease for the whole of the lease (7 days; 24 h for trials) plus grace_days. Either the service was unreachable for that long, or it stopped issuing leases: the license is disabled, past its commercial expiry, or suspended for 30 days of silence (#lease-refused, #silent-usage). The server drains for KUBEMQ_LICENSE_DRAIN_SECONDS and exits with code 3.

Fix. Look at last_refresh.outcome in GET /api/v1/license before the stop, or in the log history: unreachable means a network problem — restore egress and restart; unsigned_4xx with lease_refused means the license needs renewing — contact KubeMQ, then restart. A trial that ended converts through the reminder email.

Grace countdown

Log line: license: lease expired <time>; running on grace, <n> days left … #grace-countdown

Cause. A warning every minute. The lease's exp has passed and the service is still unreachable (or refusing), but you are inside grace_days. The server runs normally.

Fix. Treat it as an outage with a deadline: run_until in GET /api/v1/license is the moment it stops. Restore connectivity; a single successful refresh ends the countdown and issues a fresh 7-day lease.

Revoked

Log line: license: revocation confirmed (3 assertions): <reason>; draining … #revoked

Cause. The licensing service has answered three refreshes, at least 10 minutes apart, with a signed revocation for this license — the result of a deliberate "revoke now" or "enforce cap" action by KubeMQ. The server drains and exits with code 3, and will refuse to start again with the same license.

Fix. Contact KubeMQ. revocation.reason in GET /api/v1/license carries the reason given. The three confirmations are persisted in <store>/license/revocation.json, so the server refuses to boot with that license until KubeMQ reinstates it — a reinstated license is reissued with a newer token_version. To apply it, delete both <store>/license/lease.jws and <store>/license/revocation.json, update the key if it changed, and restart; the server activates again and the new lease clears the record.

Revocation seen

Log line: license: revocation assertion seen (<n> of 3): <reason> … #revocation-seen

Cause. A warning. One or two signed revocations have been received; the server now re-polls every 10 minutes and will drain after the third. A stray or replayed response cannot cause this — only a validly signed, fresh assertion counts, and an assertion is counted only when both its issue time and its arrival are at least 10 minutes after the previously counted one. The count is persisted, so a restart with one or two on record resumes the 10-minute polling with the same count.

Fix. Contact KubeMQ immediately if this is unexpected. revocation.seen_count and revocation.first_seen in GET /api/v1/license show the timeline.

Clock rollback

Log line: license: wall clock <now> is behind the persisted high-water mark <hwm> … #clock-rollback

Cause. The system clock is more than 10 minutes earlier than the latest local time this server has ever seen (persisted in <store>/license/clock.hwm). This blocks a clock set back to extend a lease. What happens next depends on the mode:

• Online. The server asks the licensing service to confirm: at boot it attempts a refresh and refuses to start if that refresh does not return a signed lease; on a running server it requests one refresh and waits for its outcome — a signed lease clears the condition, a 429 rate_limited answer defers the decision to the next scheduled refresh, and any other outcome (unreachable, refused) stops the server. A successful refresh resets the mark to the local clock — the signed lease is what validates local time; the service's server_time is used only for the skew figure.
• Offline. There is no service to ask. The server refuses to start, or stops if it is running.

Fix. Fix the clock (NTP) first; that is almost always the cause. If the clock is right and the mark is wrong (for example a store restored from a backup taken on a machine with a bad clock), stop the server, delete <store>/license/clock.hwm, and start it again; the server re-persists the mark from its local clock.

Clock skew

Log line: license: local clock differs from the licensing service by <n> s … #clock-skew

Cause. The local time and the service's time differ by more than 5 minutes. The service's time is taken only from a response that carried a signed lease: the lease's signed iat, refined by the unsigned server_time field when that agrees with the iat within 5 minutes (a server_time further off is ignored, and logged under this anchor). A warning, and clock.skew_seconds in GET /api/v1/license shows the value. Skew above 24 hours becomes a stop once the lease has expired. The same anchor is used when a usage report is answered 413 or 422 — the row is dropped as terminal, typically because its timestamps are out of range.

Fix. Synchronize the clock with NTP. Skew never extends a lease: expiry is enforced from lease exp + grace regardless.

Over cap

Log line: license: <n> active installations, license allows <max_instances> … #over-cap

Cause. A warning every minute. The lease carries over_cap: true — more installations (distinct fingerprints, or concurrent server processes) have reported than the license allows. Nothing is enforced automatically; the situation is visible to KubeMQ, who may contact you or, after notice, revoke the newest installations.

Fix. Stop the extra installations, or ask for a higher cap. Common accidental causes: containers started without a store volume (each restart is a new fingerprint), and Kubernetes pods without the kube-system permission (each pod counts separately).

Silent usage

Log line: license: no usage report accepted for 72 h … #silent-usage

Cause. A warning every minute. Leases are still being refreshed but the service has not accepted a usage report from this license for 72 hours — usually because POST /v1/usage/report is blocked while /v1/license/refresh is not, or every report is rejected. While silent, leases are shortened to 24 hours; after 30 days of silence the service stops issuing leases (503 silence_suspended) and the server runs out its lease and grace.

Fix. Check usage.last_outcome and usage.last_status in GET /api/v1/license. error means the request is not getting through — allow the same egress you allow for refresh. rejected with a 422 means a clock problem (#clock-skew). See Usage reporting for what is sent.

Store not persistent

Log line: license: store path <path> is not a mounted volume; the license cache will not survive a restart … #store-not-persistent

Cause. A boot warning. The store directory is on the same filesystem as /, which inside a container means it dies with the container. The lease cache, the instance id and the usage cursor are all under it.

Fix. Mount a volume at the store path (/kubemq/store in the official image) — -v kubemq-data:/kubemq/store on Docker, a volumeClaimTemplate on Kubernetes. See The lease cache. A throwaway run can ignore the warning.

Missing kube-system permission

Log line: license: could not read namespace kube-system (<error>); using persisted instance id … #missing-kube-system-permission

Cause. A boot warning on Kubernetes. The server's ServiceAccount lacks get on namespaces for kube-system (or the API server was unreachable), so the fingerprint is a per-pod persisted id instead of the cluster's namespace UID. fingerprint_source reads persisted.

Fix. Bind the kubemq-cluster-next-license ClusterRole to the pod's ServiceAccount — the chart does this with the kubemq-cluster-next-<namespace>-license-crb binding; the manifest is in Kubernetes without the operator. Then restart the pods. Until then every pod is a separate installation, and an offline file bound to the cluster UID will not match.

Untrusted signing key

Log line: license: the licensing service has no signing key this server trusts (422 untrusted_kid) … #untrusted-kid

Cause. The server's built-in set of trusted public keys is older than the service's current signing key. The service refuses to sign with a key the server would reject. On a first boot (no cached lease) this is an immediate refusal — there is no lease to fall back on. A running server keeps its cached lease and runs until lease + grace.

Fix. Upgrade the server to a release that ships the new key. Key rotations are announced in the release notes at least two releases ahead.

Lease refused

Log line: license: refresh answered HTTP 401 lease_refused (unsigned); keeping the cached lease until <run_until> … #lease-refused

Cause. The cached lease is valid but the refresh did not produce a new one. This anchor covers every unsigned answer to a refresh:

• 401 lease_refused — the service will not issue a lease: the license has been disabled, or its commercial expiry has passed.
• any other unsigned 4xx (last_refresh.outcome: unsigned_4xx), except 422 untrusted_kid (#untrusted-kid).
• a 5xx (unsigned_5xx) — a service-side error, except 503 silence_suspended (#silent-usage).
• a 200 whose body is not a signed lease (malformed_200) — almost always a proxy or captive portal answering in place of license.kubemq.io.

None of these is signed, so none stops the server immediately — it runs on the cached lease, retries with backoff, and drains at lease exp + grace (#lease-and-grace-exhausted).

Fix. Read last_refresh.outcome and last_refresh.status in GET /api/v1/license. For 401, compare license_expires_at with today: if it is in the past, renew with KubeMQ — the next refresh after renewal picks up the new expiry with no restart; if it is in the future, the license was disabled — contact KubeMQ. For malformed_200, check what sits between the server and the endpoint. For a 5xx, wait; the server keeps retrying.

Quick reference