Skip to content

← Production Guide · Control plane

PG-27 · 401 with a valid token?

401 with a valid token? Operational lesson: A fresh token hides the problem. Test with one older than the token cache.

Operational lesson. A fresh token hides the problem. Test with one older than the token cache.

The memo as text

Situation. openstack token issue works, then service calls fail with 401, often every other time. The Keystone node asked to validate the token cannot read it, or finds it expired or revoked.

First checks (read-only):

  1. Key checksums, on every node: md5sum /etc/keystone/fernet-keys/* (Kolla: in keystone_fernet)
  2. Is key distribution healthy?: Kolla-Ansible: health of the keystone_fernet container
  3. Which node rejects an aged token?: aged token, suspect node first: only the 1st answer counts
  4. Clocks of the Keystone nodes: chronyc tracking, or timedatectl ; date -u

Think before acting:

  • 🛑 Destructive. Pushing one node's keys to the others invalidates live tokens the others issued. Choose the source by checksum, not by key number.
  • ℹ️ Know this. A test with a fresh token proves nothing: every node answers it from the shared cache, without decrypting it.

Go deeper

This memo is a memory aid. The procedure, with every command, its risk warning and its limits, is in the runbook:

Where this holds. Written from the Keystone 2026.1 source and lab-verified on Kolla-Ansible with three controllers, where Keystone caches tokens in memcached shared by all nodes (600 s by default) and the keystone_fernet containers rotate and push the keys. Other deployments distribute keys differently. Intermittent failure is common but not the rule. If keys were replaced on every node, every client must simply authenticate again.

LinkedIn kit

openstack token issue works. But a script that keeps its token starts getting 401: sometimes every call, sometimes every other one.

Behind a load balancer, that means a Keystone node cannot validate tokens another one issued. Two usual reasons, and they do not hit the same tokens:

→ Fernet keys differ between nodes: it breaks tokens older than the token cache, 10 minutes by default. Long-running clients, scripts, Horizon sessions.
→ A clock is off on one node: it breaks fresh tokens too.

Here is the trap. On Kolla-Ansible, Keystone caches every token it issues in memcached, shared by all nodes. For 10 minutes, any node answers from the cache, without decrypting anything. So your test with a brand new token passes on every node, with diverged keys.

Test with a token older than the cache. And compare key checksums across nodes, not key numbers.

Runbook: https://github.com/OzarLand/ozarland-academy/blob/main/docs/runbooks/recovery/keystone-401-for-every-request-fernet-keys-or-clock.md
More memos: https://github.com/OzarLand/ozarland-academy/tree/main/docs/production

#OpenStack #Keystone #Authentication #ControlPlane #CloudOperations #OzarLand

Source: 401-with-a-valid-token.yml. Found an error, or does your platform behave differently? Open an issue or see CONTRIBUTING.