Keystone¶
This page is the starting point for Keystone, the keystone role: the containers keystone, keystone_ssh and
keystone_fernet (roles/keystone/defaults/main.yml:2-56), on the control nodes of the example inventory
(inventory/multinode:93-94), selected by the tag keystone (site.yml:483-495). Keystone is a hard deploy-time
dependency of the 31 roles whose tasks use service-ks-register (keystone itself included; grep of roles/*/tasks;
2023.1 had 37, the difference being the removed roles freezer, murano, sahara, senlin, solum and vitrage): registration
runs run_once (roles/service-ks-register/tasks/main.yml:125-126) with openstack_interface (internal) and
authenticates against keystone_internal_url (roles/service-ks-register/defaults/main.yml:5,
group_vars/all.yml:710, group_vars/all.yml:990-991) CODE (Component
dependencies §1). The lists below say where
the study covers how to operate it, its impact, its dependencies and the evidence, what to check around a run and what
stays open; the service card follows them.
What changed in 2024.1¶
- The admin port and admin endpoint code is gone CODE. 2024.1 defines no
keystone_admin_port,keystone_admin_listen_port,keystone_admin_urloradmin_protocol(no match underansible/). The Keystone ports are nowkeystone_public_port,keystone_public_listen_port: "5000",keystone_internal_port: "5000"andkeystone_internal_listen_port(group_vars/all.yml:444-447). Thekeystone_adminHAProxy entry that 2023.1 kept for upgrades is removed: the role defines onlykeystone_internalandkeystone_external(roles/keystone/defaults/main.yml:11-30), and Apache listens onkeystone_public_listen_portonly, with one virtual host (roles/keystone/templates/wsgi-keystone.conf.j2:10,roles/keystone/templates/wsgi-keystone.conf.j2:36). - No step removes an admin endpoint any more CODE.
keystone_service_endpointslists onlyinternalandpublic(roles/keystone/defaults/main.yml:200-202); the 2023.1adminentry withstate: absentis gone, andservice-ks-registerno longer passes astatetoos_keystone_endpoint(roles/service-ks-register/tasks/main.yml:24-46).upgrade.ymlends with the database steps (roles/keystone/tasks/upgrade.yml:1-50): the 2023.1 import ofservice-ks-registerthat removed the admin endpoint during an upgrade is not there. Anadminendpoint still present in the catalogue is therefore left as it is by 2024.1 code; whether one is present on a given platform is NOT AVAILABLE. The bootstrap wrapper is called with seven arguments, without an admin URL (roles/keystone/tasks/register.yml:4-7), which the kolla script treats as its "modern mode" (kolladocker/keystone/keystone/keystone_bootstrap.sh:19-30). - URLs are built with the
kolla_urlfilter CODE.keystone_internal_urlandkeystone_public_urlare<fqdn> | kolla_url(<protocol>, <port>)(group_vars/all.yml:960-961); the filter is registered infilter_plugins/address.py:25-30. It puts the address in URL context (IPv6 in brackets) and omits the port forhttpon 80 andhttpson 443 (kolla_ansible/kolla_url.py:20-41). The internal URL now useskeystone_internal_port; 2023.1 usedkeystone_public_portfor both URLs. keystone_public_portfollows the single external frontend option CODE: withhaproxy_single_external_frontend(defaultfalse) it becomes 443 or 80 (group_vars/all.yml:444,group_vars/all.yml:931-932), sokeystone_public_urlthen carries no port;keystone_public_listen_portstays5000(group_vars/all.yml:445). With the option on, thehaproxy-configrole also writes anexternal-frontend-mapblock per service, which notifies the HAProxy restart (roles/haproxy-config/tasks/main.yml:25-53).- Port pairing to keep in mind CODE: the internal frontend sends to
keystone_internal_listen_port(roles/keystone/defaults/main.yml:17-18) while Apache listens onkeystone_public_listen_port(roles/keystone/templates/wsgi-keystone.conf.j2:10). Both are 5000 by default (group_vars/all.yml:445-447); an override of one without the other would make the backend port differ from the listening port. What a platform has overridden is NOT AVAILABLE. - New messaging options in
keystone.confCODE: with the default quorum queues,rabbit_quorum_queue = trueis rendered (roles/keystone/templates/keystone.conf.j2:77-86,group_vars/all.yml:268; see F17). Withom_enable_queue_manager(defaultfalse,group_vars/all.yml:269) thekeystonecontainer also mounts/dev/shm(roles/keystone/defaults/main.yml:159) andkeystone.confgetsuse_queue_manager(roles/keystone/templates/keystone.conf.j2:63-68) and a[oslo_concurrency]lock path (roles/keystone/templates/keystone.conf.j2:116-119). Theprocessnameline of that block is guarded byservice_name == 'keystone-api', but the file is rendered withservice_nameset to thekeystone_serviceskeyskeystoneandkeystone-fernet(roles/keystone/tasks/config.yml:60-74), so that line is never rendered CODE. - Role assignments:
service-ks-registernow grants a role on a project, a domain or the system (roles/service-ks-register/tasks/main.yml:103-119), and the keystone role no longer passes a list of extra roles to register (roles/keystone/tasks/register.yml:14-19) CODE. - Still holds in 2024.1 CODE: the handlers are the same five, in the same order, with the module renamed
kolla_container(roles/keystone/handlers/main.yml:5-98); the play still takesserialfromkolla_serial, default0(site.yml:488), and now also setsmax_fail_percentage, default 100 (site.yml:489-492), which never stops the play early (Controlled batches §1.3).
Where to look¶
Operate¶
- Scenario: Reconfigure a single service §2, Reconfigure a single service §3
- Command: Reconfigure a single service §1.1
- Membership: Change control-plane membership: adding or removing a keystone host changes the Fernet rotation schedule and the push targets on every keystone host.
Impact and risk¶
- What the tag executes (
keystone): Tags §3, row<svc>; Tags §3.1, row<svc>API service - Risks: R19, R8, R24
Dependencies¶
- Dependencies: Component dependencies §1, Component dependencies §3
Evidence¶
- Fact sheet: fact sheet B §3
- Image side: Kolla images
§4 (row
keystone), Kolla images §4.1, Kolla images §5 - Also: handlers: Tags §5
Before and after a run¶
- Tags §3.1, row
<svc>API service: safe on its own "yes, it is the intended unit; it still runs Keystone registration and DB sync"; batch withANSIBLE_SERIAL. Prerequisites: MariaDB, RabbitMQ, memcached, Keystone (registration), HAProxy VIP. - R19, precautions for a Keystone
reconfigure:ANSIBLE_SERIAL; verify fernet key consistency before and after. - With the default serial 0, a reconfigure with a config change restarts
keystone_ssh, thenkeystone_fernet, thenkeystone, in the handler definition order (roles/keystone/handlers/main.yml:29-75), each on all keystone hosts of the play at once (site.yml:488) CODE. SetANSIBLE_SERIALfor a rolling reconfigure. The details are in fact sheet B §3.9. ANSIBLE_SERIALhas a limit on a first deployment: when no host of the run so far was found with a runningkeystone_fernetcontainer and some keystone hosts are outside the current batch, the role fails on purpose rather than bootstrap the Fernet keys (roles/keystone/tasks/bootstrap_service.yml:22-31) CODE.- R8, precautions for a HAProxy config change:
ANSIBLE_SERIAL=1for haproxy changes;--skip-tags haproxywhen you don't intend to change frontends. - R24, container stop timeout: in 2024.1 the stop goes through the container's systemd
unit, whose
ExecStopis/usr/bin/<engine> stop <name> -t <graceful_timeout>(module_utils/kolla_systemd_worker.py:32), withdocker_graceful_timeout60 s (group_vars/all.yml:139) (F3). Precautions: drain long operations before restarts; raisedocker_graceful_timeoutif needed. The first 2024.1 check of a container whose unit 2023.1 wrote reports drift (F16). - R22, precautions for an inventory reordering (the keystone group is one of the groups
it names): treat inventory order as configuration; never reorder casually. For Keystone the order matters because the
Fernet rotation schedule of each host is computed from its index in
groups['keystone'](roles/keystone/tasks/config.yml:195-200) CODE. - Reconfigure a single service
§3,
verification of
-t <svc> reconfigure: before,kolla-ansible -t <svc> genconfig, thenkolla_set_configs --checkin each<svc>container and a diff of/etc/kolla/haproxy/services.d/<svc>*.cfgagainst the previous version; after,openstack endpoint list --service <name>is unchanged for each Keystone service the role registers, the API answers through the VIP, the container start times show which hosts restarted, and the haproxy container's start time shows whether HAProxy restarted. For Keystone, the endpoint list has aninternaland apublicendpoint only (roles/keystone/defaults/main.yml:200-208); anadminendpoint left from an older release is not removed by 2024.1 code (see What changed in 2024.1).
Known limits and open points¶
- Known bugs and limitations, row
keystone: fernet cron generator hour calculation; precheck vs generator expiry limits; push script stops at the first unreachable peer. Status: all three still hold in 2024.1 CODE, and the files involved are unchanged from 2023.1: the generator computeshour = time % HOUR_SPAN(roles/keystone/files/fernet_rotate_cron_generator.py:69); the precheck assertsfernet_token_expiry <= 604800(roles/keystone/tasks/precheck.yml:43-53) while the generator is fedfernet_key_rotation_interval(roles/keystone/tasks/config.yml:195-200);fernet-push.shruns underset -o errexitand rsyncs to each peer in turn (roles/keystone/templates/fernet-push.sh.j2:3-4,roles/keystone/templates/fernet-push.sh.j2:17-22). - The key distribution after a deploy or reconfigure is one
fernet-push.shrun,run_once, on the first host wherekeystone_fernetwas found running, or on the first keystone host when none was (roles/keystone/tasks/distribute_fernet.yml:12-18) CODE. - Q11: "Fernet key distribution failure modes". Status: NOT AVAILABLE.
- What is not
known, row
"Keystone": the sources do not establish whether authentication, and so the other services' API requests, fails while
keystone restarts or is down (R19); Fernet key distribution failure modes
(Q11); whether Keystone still reads a removed policy file after the
restart; the upstream semantics of
keystone-manage bootstrap; whether thekeystone_fernetcontainer restarts when key 0 is missing. - What is not known:
the sources do not establish whether the SSH clients accept the new
nova_sshandkeystone_sshhost keys after a recreate without failing. - F5, "
reconfigure=deploy": a reconfigure re-runs Keystone registration, DB creation/grants and the DB-sync bootstrap containers of most services CODE. For Keystone, the deploy flow runs config, check-containers, bootstrap, the handlers, key distribution, then registration (roles/keystone/tasks/deploy.yml:1-21). - fact sheet B §3.9, "Operationally notable behaviour, risks and oddities" of the keystone role.
- fact sheet B §3.10: NOT AVAILABLE: whether the API services whose
[keystone_authtoken]sections point keystonemiddleware at Keystone fail to authenticate requests while Keystone or memcached is unavailable. - Kolla images §5, row
keystone-fernet: without key 0 on the node the start waits up to 10 × 60 s and the script exits 1 before the cron command is exec'd, so no rotation cron runs there meanwhile (roles/keystone/templates/fernet-node-sync.sh.j2:15-28) CODE; whether the container then restarts is NOT AVAILABLE. - Kolla images §6, item 18: SSH host keys of
nova_sshandkeystone_sshchange on every recreate, because/etc/sshis on no volume (roles/keystone/defaults/main.yml:36-41); the keystone SSH client setsStrictHostKeyChecking noandUserKnownHostsFile /dev/null(roles/keystone/templates/ssh_config.j2:2-3) CODE; whether the SSH client then accepts the new keys without failing is NOT AVAILABLE (OpenSSH semantics). - Kolla images §7: not available from the
sources: the upstream semantics of
keystone-manage bootstrap, in particular whether it resets the admin password tokeystone_admin_password.
Service card¶
The rows of the card are explained on the Components page.
| Item | Value |
|---|---|
| Play | site.yml:483 |
| Tags | keystone |
| Play host patterns | keystone |
| Node types in play (example inventory) | control |
| Node types running its containers | control |
| Containers | keystone, keystone_ssh, keystone_fernet |
deploy/reconfigure flow (tasks/) |
config → check-containers → clone (if) → bootstrap → flush handlers → distribute_fernet → register → register_identity_providers (if) |
| Task files | bootstrap.yml, bootstrap_service.yml, check-containers.yml, clone.yml, config-federation-oidc.yml, config.yml, config_validate.yml, copy-certs.yml, deploy-containers.yml, deploy.yml, distribute_fernet.yml, loadbalancer.yml, main.yml, precheck.yml, pull.yml, reconfigure.yml, register.yml, register_identity_providers.yml, stop.yml, upgrade.yml |
| Templates | crontab.j2, fernet-healthcheck.sh.j2, fernet-node-sync.sh.j2, fernet-push.sh.j2, fernet-rotate.sh.j2, keystone-fernet.json.j2, keystone-ssh.json.j2, keystone-startup.sh.j2, keystone.conf.j2, keystone.json.j2, ssh_config.j2, sshd_config.j2, wsgi-keystone.conf.j2 |
| Handlers (definition order) | Init keystone database upgrade; Restart keystone-ssh container; Restart keystone-fernet container; Restart keystone container; Finish keystone database upgrade |
| Key variables | enable_keystone, keystone_token_provider, fernet_token_expiry, fernet_key_rotation_interval |
| Role dependencies | none |
Related¶
- Reconfigure a single service —
-t keystone reconfigure, the command example of the page: scope, restarts and one-shot - Change control-plane membership — what adding or removing a controller changes, Keystone's Fernet schedule included
- HAProxy + Keepalived — the loadbalancer play renders the keystone frontends under
-t keystone(site.yml:203-207), and HAProxy restarts if they changed - Memcached — Keystone's token cache (
roles/keystone/templates/keystone.conf.j2:48-51) - Component dependencies §1 — Keystone in the shared substrate that every OpenStack service uses
- fact sheet B §3.10 — the runtime dependency of other services on Keystone, counted by role
- Pre-execution checklist — the checks before every production run
- Components — the other component pages