kolla-ansible deploy fails in the keystone role: Fernet bootstrap, key distribution, or key 0 missing¶
Symptom¶
One of these, during kolla-ansible deploy, reconfigure or upgrade, in the keystone role:
TASK [keystone : Fail if any hosts need bootstrapping and not all hosts targeted] ****
[ERROR]: Task failed: Action failed: Some hosts (controller-1) need Fernet key bootstrapping, but not all Keystone hosts are in the target list. ...
fatal: [controller-1]: FAILED! => {"changed": false, "msg": "Some hosts (controller-1) need Fernet key bootstrapping, but not all Keystone hosts are in the target list. Stopping as it may be unsafe to proceed. Please run without --limit or --serial to bootstrap these hosts.\n"}
TASK [keystone : Waiting for Keystone SSH port to be UP] ****
ok: [controller-2]
ok: [controller-1]
FAILED - RETRYING: [controller-3]: Waiting for Keystone SSH port to be UP (10 retries left).
TASK [keystone : Run fernet key distribution] ****
fatal: [controller-1]: FAILED! => ... "cmd": "docker exec -t keystone_fernet /usr/bin/fernet-push.sh" ...
or the keystone container restarts, and its log shows:
ERROR: Fernet keys have not been populated, rechecking in 5 seconds
...
CRITICAL: Waited for 3 minutes - failing
The task names and messages are those of ansible/roles/keystone/tasks/bootstrap_service.yml,
distribute_fernet.yml and templates/keystone-startup.sh.j2 in Kolla-Ansible 2026.1. The surrounding Ansible
output is reconstructed, not copied from a run, except the bootstrap message, the RETRYING line and the
keystone log lines, which were captured on a lab (see Lab observations). The bootstrap message names only the
hosts in the run.
Impact¶
Keystone issues and validates every token. Without Fernet keys on a Keystone host, that host's API fails; with keys
that differ between hosts, a token issued by one host is rejected by another, which shows as random 401 errors
behind the load balancer. Running instances are not affected.
Quick checks¶
On each Keystone host, read-only:
# 1. Containers: keystone, keystone_fernet, keystone_ssh
sudo docker ps -a --format '{{.Names}}\t{{.Status}}' | grep -E '^keystone'
# 2. The keys: key 0 must exist, owned by keystone:keystone; compare file names and hashes between hosts
sudo docker exec keystone_fernet ls -l /etc/keystone/fernet-keys/
sudo docker exec keystone_fernet sh -c 'sha256sum /etc/keystone/fernet-keys/*'
# 3. The container's own check: key 0 present and a dry-run rsync to the other hosts
sudo docker exec keystone_fernet /usr/bin/fernet-healthcheck.sh; echo "exit=$?"
Between Keystone hosts, read-only:
# 4. The keystone_ssh port (keystone_ssh_port, 8023 by default) on the api_interface addresses
nc -zv -w 3 <other-controller-api-address> 8023
Logs: docker logs keystone, docker logs keystone_fernet, docker logs keystone_ssh, and
/var/log/kolla/keystone/.
Diagnosis¶
How Kolla-Ansible 2026.1 handles Fernet keys, from the keystone role:
- Bootstrap runs once, on the first Keystone host, only if no host runs a
keystone_fernetcontainer: it starts a one-off container that runskeystone-manage fernet_setup. If some hosts need bootstrapping and the run does not include every Keystone host (--limit, or a batch withkolla_serial), the role stops on purpose, because, in its own words, bootstrapping keys on an existing cluster "would overwrite existing keys, and invalidate tokens created from them". - Distribution: the role waits for the
keystone_sshport on each host (10 tries, 5 seconds apart; each host checks its ownapi_interface_address, and each try can last up towait_for's default timeout of 300 s), then runsfernet-push.shonce, inkeystone_fernet, on the first Keystone host in inventory order whosekeystone_fernetcontainer was running when the role checked (keystone_fernet_running_True), or the first Keystone host if none was. The script does anrsync --deleteof/etc/keystone/fernet-keys/over SSH to every other Keystone host, as userkeystone, onkeystone_ssh_port, with the key pairkeystone_ssh_keyfrompasswords.yml. The source is chosen by its running container, not by whether it holds keys: if that host's key directory is empty, the push empties every other host's. The keys live in thekeystone_fernet_tokensvolume, shared by thekeystoneandkeystone_fernetcontainers. - Start-up: the
keystonecontainer waits up to 3 minutes for key0before starting its API, then fails. - Health: the Docker health check of every
keystone_fernetcontainer requires key0owned bykeystone:keystoneand a successful dry-runrsyncto every other host, so anunhealthykeystone_fernetindocker psis the quickest pointer to cause 2 or 3. - Rotation later runs from cron inside
keystone_fernet, everyfernet_key_rotation_intervalseconds (by defaultfernet_token_expiryplus the allowed expired window);fernet-rotate.shends with the samefernet-push.sh, so a host with an empty key directory and a runningkeystone_fernetcan also empty the others at its next rotation.
Possible causes¶
Cause 1: a run with --limit or kolla_serial when no keystone_fernet runs anywhere¶
First deployment with a limit, or every keystone_fernet container stopped (after a kolla-ansible stop, or a
power loss) followed by a limited run. The role's code notes that it does not detect a stopped container or a volume
without a container as "bootstrapped". Tell it apart: the bootstrap message above.
Cause 2: the keystone_ssh path between controllers is broken¶
The keystone_ssh container is not running, port 8023 is blocked between controllers, or the SSH key pair differs
between hosts (different passwords.yml used for different runs). Tell it apart: the SSH port wait fails, or
fernet-push.sh fails with an SSH or rsync error; fernet-healthcheck.sh exits non-zero.
Cause 3: key 0 is missing or has the wrong owner on a host¶
The keystone_fernet_tokens volume of a host was removed or emptied (a reinstalled controller), and distribution
has not reached it yet. Tell it apart: ls -l /etc/keystone/fernet-keys/ on that host is empty or not owned by
keystone:keystone; the keystone container logs the waiting messages above.
Resolution¶
For cause 1¶
Run the command on every Keystone host, without --limit and without kolla_serial, so that the role can decide
safely. If every keystone_fernet was only stopped and the keys still exist in their volumes, start the containers
first, so that the role sees them running and does not bootstrap. Check each host's keys first (read-only; the
container is stopped, so look at the volume):
sudo ls -l /var/lib/docker/volumes/keystone_fernet_tokens/_data/ # expected: files 0, 1, ... owned by the keystone user (42425 on the host)
# with Podman: /var/lib/containers/storage/volumes/keystone_fernet_tokens/_data/
⚠️ State-changing. Starts the stopped
keystone_fernetcontainer of a host; nothing else. Start it only on hosts whose key directory is not empty: the first running one becomes the source of the next push, which deletes on the other hosts whatever it does not have. Undo:sudo systemctl stop kolla-keystone_fernet-container.service.
# on each Keystone host whose key directory holds keys, if the container exists but is stopped
sudo systemctl start kolla-keystone_fernet-container.service # Kolla-Ansible 2026.1: systemd units, not docker start
🛑 Destructive. A full run of the keystone role on every host. If no
keystone_fernetruns anywhere, it runsfernet_setupon the first Keystone host: on an empty key repository this creates new keys, every existing token becomes invalid and users must authenticate again (whetherfernet_setupreplaces keys already present was not checked for this page). If the push source has an empty key directory, the keys of every other host are deleted and every Keystone API fails. Undo: none for the invalidated tokens or the deleted keys. Before running it, copy/etc/kolla/keystone/fernet-keys(or thekeystone_fernet_tokensvolume) from a host whose keys are complete, so the keys can be put back.
kolla-ansible deploy -i <inventory> --tags keystone
For cause 2¶
Start keystone_ssh where it is down, open keystone_ssh_port between the controllers on the API network, and use
the one passwords.yml of the deployment. Then run the keystone role again (above, the same ⚠️ applies), which
redistributes the keys.
For cause 3¶
Make sure the push will come from a host that has the keys, then let the role redistribute them.
-
On the host with the missing keys, read-only:
sudo docker exec keystone_fernet ls -l /etc/keystone/fernet-keys/(or the volume path above if the container is stopped). If the directory is empty and that host'skeystone_fernetis running, stop it, so that it cannot be the push source, neither in the deploy nor at its next cron rotation:⚠️ State-changing. Stops
keystone_ferneton the host with no keys: its rotation and pushes stop; itskeystoneAPI was already failing without keys. The deploy below starts it again. Undo:sudo systemctl start kolla-keystone_fernet-container.service.sudo systemctl stop kolla-keystone_fernet-container.service -
Check that at least one other Keystone host has a running
keystone_fernetand a full key directory (thesha256sumcommand of the quick checks). - Run the keystone role on every host:
kolla-ansible deploy -i <inventory> --tags keystone(the same ⚠️ as above). The push now comes from the first host, in inventory order, whosekeystone_fernetwas running (the host stopped in step 1 is not among them, even though the deploy starts it again: the role chooses the source before it restarts containers). - Compare the key hashes on every host (Verification).
Do not copy keys by hand between hosts, and never run fernet_setup yourself on one host of a running cluster: that
host's keys would no longer match the others'.
Verification¶
# on each Keystone host: the same key files, with the same hashes
sudo docker exec keystone_fernet sh -c 'sha256sum /etc/keystone/fernet-keys/*'
sudo docker exec keystone_fernet /usr/bin/fernet-healthcheck.sh; echo "exit=$?" # exit=0
# from a client, several times (the load balancer spreads the calls over the hosts)
openstack token issue
openstack project list
Lab observations¶
OpenStack 2026.1, Kolla-Ansible 22.2.0 (ansible-core output format), 3 controllers with Keystone, 2026-10-27.
- Cause 1:
keystone_fernetstopped withsystemctlon the three controllers, thenkolla-ansible deploy --tags keystone --limit controller-1: failed in 49 s at Fail if any hosts need bootstrapping and not all hosts targeted, message as shown in Symptom (only the targeted host named). No key was touched. The volume paths showed0and1owned by42425;systemctl starton the three hosts made themhealthy. - Cause 2, simulated with an
iptablesDROP of port 8023 on controller-3:fernet-healthcheck.shon controller-1 exited 1 andnctimed out. The fulldeploy --tags keystonethen sat in Waiting for Keystone SSH port to be UP: threeFAILED - RETRYINGlines for controller-3 in about 18 minutes, consistent with 300 s per try (10 retries would take about 55 minutes before the failure). After the rule was removed, the next try passed, the distribution ran, and the deploy ended with no failure. - Cause 3: the key directory of controller-3 emptied,
keystonerestarted there:ERROR: Fernet keys have not been populated, rechecking in 5 seconds, thenCRITICAL: Waited for 3 minutes - failingand the container exited; tokens kept working through the other two hosts. Steps 1 to 4 ran as written (withsystemctl stop): the deploy took 1.5 minutes, the push ran from controller-1, and the three hosts ended with the same two key hashes,fernet-healthcheck.shexit 0, andtoken issue/project listworking six times out of six. - Not run: a first deployment with
--limit, a run withkolla_serial, a bootstrap on an empty cluster, a push from an empty host, a differentkeystone_ssh_key, andkeystone_sshstopped.
Validity and limits¶
- Written from Kolla-Ansible 2026.1 code (
ansible/roles/keystone/tasks/bootstrap_service.yml,distribute_fernet.yml,config.yml,templates/fernet-*.sh.j2,templates/keystone-startup.sh.j2,defaults/main.yml,ansible/group_vars/all/keystone.yml) and the Keystone Fernet FAQ. Reproduced on a lab for the three causes, as described in Lab observations. sha256sumof the key files is used to compare hosts without printing the keys; never paste the keys anywhere.- Not covered: rotating
keystone_ssh_key(see the password rotation guide), federation, Keystone behind an external load balancer, and Podman.
Related¶
- Rotating the secrets in passwords.yml
- deploy, reconfigure, upgrade and the others: what
--limitandkolla_serialdo - Kolla-Ansible 2026.1: Keystone guide
- Keystone 2026.1: Fernet token FAQ
Provenance: written by agent:ozarland-author, from the Kolla-Ansible 2026.1 code and documentation and the Keystone documentation. Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your topology, and what happened.