Every API call fails at once. Expired TLS certificates or inconsistent Keystone Fernet keys¶
Two components of the control plane age silently until they fail completely, and when they fail, nothing works: no VM creation, no listing, no console, often no Horizon. Running instances are unaffected. Both failures are dated in advance and entirely preventable, which is why they are embarrassing and why they belong on the first screen of any monitoring.
Symptom¶
- Certificates: every client gets a TLS error (
certificate verify failed,certificate has expired) or the load balancer marks backends down because its health checks fail TLS. Services that talk to each other over TLS (Nova to Neutron, Keystone to the database or RabbitMQ when TLS is enabled there) log SSL errors. The failure starts at an exact timestamp. - Fernet keys:
openstack token issueworks on some Keystone nodes and not others, or tokens just issued are rejected (401 Unauthorized,The request you have made requires authentication) by the next service in the chain. Keystone logsCould not recognize Fernet tokenor key-loading errors (on 2026.1 at the default log level:Invalid user tokenthenAuthorization failed. The request you have made requires authentication.). It starts right after a key rotation or a Keystone node rebuild, or later: Keystone and the services cache token validations in memcached, so a token already validated keeps working until its cache entry expires.
Impact¶
Complete loss of the control plane for users and automation. Running instances keep running. Monitoring based on the APIs goes blind at the same moment, so the alert often comes from users.
Quick checks¶
# Expiry of the certificate behind each endpoint (repeat for internal and public VIPs, ports 5000, 8774, 9696, 9292, 8776, 8780...)
openssl s_client -connect <internal-vip>:5000 </dev/null 2>/dev/null | openssl x509 -noout -enddate -subject -issuer
# RabbitMQ and MariaDB if TLS is enabled there
openssl s_client -connect <controller>:5671 </dev/null 2>/dev/null | openssl x509 -noout -enddate
openssl s_client -connect <controller>:3306 -starttls mysql </dev/null 2>/dev/null | openssl x509 -noout -enddate
# Fernet keys identical on every Keystone node (same files, same checksums)
# Kolla-Ansible: sudo docker exec keystone ls -l /etc/keystone/fernet-keys
# sudo docker exec keystone sh -c 'md5sum /etc/keystone/fernet-keys/*' # the glob needs a shell
# Any: ls -l /etc/keystone/fernet-keys && md5sum /etc/keystone/fernet-keys/*
# Compare across nodes. Key "0" is the staged key, the highest number is the primary.
# Does a token issued on one node validate on another?
T=$(openstack token issue -f value -c id)
curl -s -o /dev/null -w "%{http_code}\n" -H "X-Auth-Token: $T" -H "X-Subject-Token: $T" <scheme>://<other-keystone-node>:5000/v3/auth/tokens # 200 or 401
On an endpoint without TLS, the openssl checks print Could not read certificate from <stdin> and Unable to load
certificate: there is no certificate on that port, not an expired one.
Diagnosis¶
Certificates. The chain of trust has several links, each with its own expiry: the endpoint certificates on
the load balancer, the backend certificates if TLS terminates on the services, the CA certificate that every
service and every agent trusts ([DEFAULT] cafile, system CA bundle), and the client certificates used for
RabbitMQ or MariaDB when mutual TLS is on. Check each; an expired CA breaks everything at once, an expired backend
certificate breaks one service. On Kolla-Ansible, kolla-ansible certificates generates and kolla-ansible deploy
or reconfigure distributes them; look at the paths in /etc/kolla/certificates/.
Fernet keys. Keys do not expire, but rotation must be done correctly and the result must be identical on every
Keystone node. A rotation that removes a key still needed to validate tokens in flight, or that leaves nodes with
different key sets, breaks authentication for the whole platform at once. keystone-manage fernet_rotate must run
on one node and the directory must then be distributed to the others (Kolla-Ansible does this with its
fernet_token_expiry and rotation schedule inside the keystone_fernet container). A rebuilt node that generated
its own keys with fernet_setup instead of receiving the shared set is the other classic case.
Possible causes¶
| Cause | How to tell |
|---|---|
| Endpoint or CA certificate expired | openssl x509 -enddate in the past; failure started at that instant |
| Certificate renewed but not distributed to agents or backends | Load balancer fine, services log SSL errors to each other; one node or service differs |
| Fernet keys differ between Keystone nodes | Checksums differ; tokens validate only on some nodes |
| Rotation removed a key still in use | Fails right after rotation; recovers for new tokens, old ones rejected |
What to do next¶
The checks on this page are read-only. Fixing is a deployment-tool procedure: regenerate and distribute certificates, or distribute one consistent Fernet key set to every Keystone node, with the tool you deployed with. For Fernet keys, follow requests fail with 401: Fernet keys differ between Keystone nodes, which chooses the source node first and pushes from it.
On Kolla-Ansible, Fernet keys live in the keystone_fernet_tokens volume on each controller, and
kolla-ansible reconfigure -t keystone re-runs the Keystone deployment that distributes them: it copies the key set
of the first host of the keystone group where keystone_fernet runs (fernet-push.sh, run once), not of a node
you choose.
🛑 Destructive. The push replaces the key set on every other Keystone node with that host's set (
rsync --delete): if that host has the wrong set, it replaces the good one everywhere, and every live token issued with the lost keys gets a 401; users and scripts must authenticate again, and builds, attachments and migrations in progress can fail. It also re-renders the Keystone configuration and runs the Keystone bootstrap, so it can restart thekeystone,keystone_fernetandkeystone_sshcontainers, with failed requests while they restart. Precondition: you have compared the key sets of every node and the first host of the group holds the right one; a quiet window; every node's key directory saved first. There is no real undo: restoring the saved sets brings the divergence back.
kolla-ansible reconfigure -i <inventory> -t keystone
Keystone reads its key repository each time it encrypts or decrypts a token, so a new key set needs no Keystone
restart. For certificates on Kolla-Ansible, kolla-ansible certificates then reconfigure; after the fix, every
service and agent that cached the old certificate may need a restart (certificate part not run on the lab). For
Fernet keys alone, the services did not need a restart on a lab: their cached service token got one 401, then they
re-authenticated by themselves.
Prevention, which is the actual runbook¶
- Put the expiry date of every certificate on the control plane on the first monitoring screen, with an alert at 30 days. Include RabbitMQ, MariaDB, Ceph dashboards and anything else behind TLS, not only the OpenStack VIPs.
- Put certificate rotation in the maintenance calendar, and rehearse it on a lab: a rotation that has never been rehearsed is a change with an unknown blast radius.
- Monitor the age and consistency of the Fernet keys across Keystone nodes. A platform that has never rotated its keys has never tested that it can.
- Include both in recovery exercises.
Lab observations¶
Kolla-Ansible 2026.1 all-in-one (one Keystone node), no TLS. The Fernet key set was replaced by a new one made with
keystone-manage fernet_setup, as a rebuilt node would do.
- Right after the swap, and after a Keystone restart, a token issued before kept validating (200) on Keystone, Nova,
Neutron and Glance: the validation was cached in memcached. After a memcached restart, the same token got 401 on
all four;
openstack token issueand new commands worked. - Nova and the scheduler got one 401 from Placement each with their cached service token, then re-authenticated; an
instance created right after reached
ACTIVEin 22 s. - Restoring the saved key set and restarting Keystone made the old token valid again (tokens issued with the wrong
set were then invalid).
kolla-ansible reconfigure -t keystonethen ran in 1 min 40 s and left the keys unchanged. - The certificate part was not tested: no TLS on that lab.
Validity and limits¶
Derived from an operator's account of long-lived platforms where certificate expiry was one of the most common causes of a complete control-plane outage. The Fernet part was confirmed on a lab (OpenStack 2026.1, Kolla-Ansible all-in-one, one Keystone node); the certificate part and the differences between Keystone nodes were not. Paths and commands are given for Kolla-Ansible; other deployment tools manage certificates and Fernet keys their own way.
Related¶
- VM creation fails platform-wide: check the shared dependencies first.
- Source: OpenStack, the Day After Tomorrow (S. Zaouam), chapters 8 and 17.
Provenance: written by agent:ozarland-author, from the book OpenStack, the Day After Tomorrow. Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your deployment method, and what happened.