Reconfigure a single service¶
Impact Restarts on drift
What kolla-ansible -t <svc> reconfigure does for one API service, with keystone as the command example: after the
gather-facts and group_by plays, which run on all hosts, the haproxy play runs on the haproxy hosts (the service's
frontend, and a HAProxy restart if the rendered frontend changed) and the service play on the service hosts (common,
then the service's containers, all of which restart on every drifted host). The page also says what a narrow tag touches
besides the service.
The scenario sections are written for the assumed deployment; the page uses the conventions of the command sections, the definition of drift and the notion of code guarantees and environment dependence.
Warning
R8: a changed services.d/<svc>*.cfg restarts HAProxy on all haproxy hosts — see
§1.2 and
§3.
1. Commands¶
1.1 Reconfigure a single service¶
kolla-ansible -i multinode -t keystone reconfigure
- Scope:
- haproxy play: keystone frontend only, on haproxy hosts;
- keystone play:
commonon keystone hosts, then config, check-containers, bootstrap (db sync), a handler flush (restarts happen here), Fernet key distribution, Keystone registration and the sanity check (roles/keystone/tasks/deploy.yml; the sanity check is off by default,kolla_enable_sanity_checks). Unlike glance, placement, neutron and nova, keystone registers last.
- Restarts:
- haproxy, on all haproxy hosts, only if the frontend changed;
- per host (loop-notify): on each keystone host where any keystone container drifted, all keystone containers of
that host restart (
keystone, pluskeystone_sshandkeystone_fernetwith Fernet tokens). Hosts without drift do not restart. WithoutANSIBLE_SERIAL, drifted hosts restart in parallel, so a change common to all hosts (globals,keystone.conf, a new image) restarts keystone on all keystone hosts at once.
- One-shot: keystone db sync. Fernet keys are re-distributed.
- Impact: if restarted, keystone restarts on all drifted hosts at once; NOT AVAILABLE whether authentication fails meanwhile, and for how long (Risk register R19).
1.2 Avoid touching HAProxy during a service run¶
kolla-ansible -i multinode -t nova --skip-tags haproxy reconfigure
--skip-tags haproxydrops the whole haproxy play, including the per-service frontend includes.- Use it when frontends are not meant to change. Otherwise a changed
services.d/<svc>*.cfgrestarts HAProxy on all haproxy hosts.
2. Execute a service-specific tag: implicit effects¶
Even with a narrow tag, the code guarantees that the run also touches:
| Implicit effect | Mechanism | Evidence |
|---|---|---|
| HAProxy frontends of the service, on all haproxy hosts, and a HAProxy restart if the rendered frontend changed | haproxy play includes <svc>/ under the same tag; haproxy-config is tags: always; handlers ignore tags |
site.yml:126-392; fact sheet A §1.9; Tags §2.2–2.3 |
common on every host of the service play (fluentd/cron/toolbox config + container check, restarts if drift) |
meta dependency inherits the tag; first-play rule | roles/, roles/; Tags §2.1 |
| Keystone registration (service, endpoints, user, role) | register.yml in deploy.yml/reconfigure.yml |
service-ks-register |
DB creation/grants and DB schema sync for most services (Neutron's DB sync: deploy only by default, Reconfigure Neutron §2) |
bootstrap.yml imported by deploy.yml |
CODE 11 one-shot container runs on a no-change redeploy of this service set (cell0 mapped; MariaDB, RabbitMQ and Fernet keys already bootstrapped): 8 DB-sync bootstraps (keystone, glance, placement, nova API, nova cell, neutron, cinder, heat) + map_cell0 + 2× list_cells, counted from each role's bootstrap_service.yml (nova: Tags §3) |
| RabbitMQ user/vhost (nova-cell only) | service-rabbitmq |
nova-cell/ |
Global MariaDB variable toggle (-t glance) |
log_bin_trust_function_creators 1→0 |
fact sheet C §1 |
Fernet key redistribution (-t keystone) |
distribute_fernet.yml |
fact sheet B §3.6 (re-run on reconfigure: §3.7) |
Actions on hosts outside --limit |
delegate_to to groups[...][0]: Neutron DB bootstrap on neutron-server[0], cell list/discovery on conductor[0], RabbitMQ user creation on rabbitmq[0] |
roles/, bootstrap_service.yml:19-20; roles/, wait_discover_computes.yml:42; roles/; Add a compute node §2 |
SIGHUP to every nova-scheduler (-t nova/nova-api) |
refresh play | nova.yml:140-156 |
What a service tag does not do: it never touches MariaDB, RabbitMQ or memcached containers (other plays), and it
never touches OVS (unless -t openvswitch). Under -t haproxy or a no-tag run, HAProxy and keepalived containers
are also checked; a service tag touches only its frontend file.
CODE -t haproxy reconfigure on an unchanged cloud: common + haproxy role on the 2 haproxy hosts
(site.yml:126-136, inventory/multinode:91-92). All 49 frontend include_role tasks are evaluated; 12 run
(placement's, which has no when, and the 11 whose enable_<svc> is true for this service set) and the other 37 are
skipped by when: enable_<svc> (site.yml:138-389). Restarts follow only a task that reports changed (Tags
§2.4); NOT AVAILABLE whether
any task does on an unchanged cloud.
3. Sequence, impact, risks and verification of -t <svc> reconfigure¶
For an API service such as keystone, glance, cinder, heat or placement:
- The gather-facts and group_by plays run on all hosts (
tags: always). - haproxy play, on the haproxy hosts: only the
<svc>include runs and rendersservices.d/<svc>*.cfg.common(a meta dependency of the haproxy role) and the haproxy role's own tasks do not carry the<svc>tag, so they are skipped CODE (Tags §2.3). HAProxy restarts on all haproxy hosts at once if that file changed (R8). - Service play, on the service hosts:
common→ Keystone registration (run_once) → config → check-containers → DB/user creation and the DB-sync container on the first host → flush. Every drifted host restarts all of the service's containers (loop-notify), all such hosts in parallel unlessANSIBLE_SERIALbatches the play. Keystone is the exception: it registers last (§1.1).
- Interruption: the service API is unavailable while its containers restart (up to 10 s stop, then start) on all its hosts at once. Add the HAProxy restart window if the frontend changed.
- VM impact: these services restart none of the containers that Tags
§3.1 traces to running VMs
CODE; any other effect on VMs is NOT AVAILABLE. In-flight glance or cinder transfers are SIGKILLed after 10 s
(R24). Of the tags in Tags §3.1, the ones that can restart those containers are
nova,nova-cell,neutron,openvswitch,ovn,iscsiandbaremetal(run bybootstrap-servers, notreconfigure; Docker restart, R14) CODE; what those restarts do to running VMs is NOT AVAILABLE. Running on computes is not the test:chronyruns on every compute but restarts none of those containers. - Risks: R8 (HAProxy restart), R19 (Keystone),
R20 (glance file backend), R23 (
--limiton a large inventory), R24 (10 s SIGKILL). - Verification:
- before:
kolla-ansible -t <svc> genconfig, thendocker exec -u root <c> /usr/local/bin/kolla_set_configs --check; echo $?in each<svc>container (without-u rootit exits 2 in images that run as a service user, e.g. glance-api; Kolla images §3.4) and a diff of/etc/kolla/haproxy/services.d/<svc>*.cfgagainst the previous version; - after:
openstack endpoint list --service <name>is unchanged for each Keystone service the role registers (eachnamein<svc>_ks_services, e.g.cinderv2andcinderv3for cinder,heatandheat-cfnfor heat), the API answers through the VIP, the container start times show which hosts restarted, anddocker ps --filter name=haproxyshows whether HAProxy restarted.
- before:
Related¶
- Risk register — the risks this page names: R8 (HAProxy
restart), R19 (Keystone), R20 (glance file backend),
R23 (
--limiton a large inventory), R24 (10 s SIGKILL) and R14 (Docker restart bybootstrap-servers) - Keystone — service card and where to look for the service of the §1.1 example
- HAProxy + Keepalived — service card and where to look for HAProxy, including the haproxy play under a service tag
- Pre-execution checklist — the checks before every production run
- Predict and inspect a run —
genconfigand the exit codes ofkolla_set_configs --check, used in the verification of §3 - Tags §2 — the Ansible 2.9 semantics behind the implicit effects: tag inheritance, handlers that ignore tags, the haproxy play, loop-notify
- Command quick lookup by tag — hosts touched, possible restarts and one-shot containers per tag
To remember
Lumo: if you keep only 5 things from this page, keep these.
- A changed
services.d/<svc>*.cfgrestarts HAProxy on all haproxy hosts at once; add--skip-tags haproxywhen frontends are not meant to change. - On each drifted host all the service's containers restart, all such hosts in parallel unless
ANSIBLE_SERIALis set: a change common to all hosts then restarts the service on all its hosts at once. - Even a narrow tag also touches the service's HAProxy frontends, Keystone registration, DB sync and
commonon every host of the service play. - The service API is unavailable while its containers restart (up to 10 s stop, then start) on all its hosts at once.
- Before:
-t <svc> genconfig, thenkolla_set_configs --checkwith-u rootin each container and a diff of theservices.dfile.
What would you do?
Lumo: a situation you could meet on a real platform, and three ways to react. Pick the one you would choose: I will tell you what this page says about it. Nothing is recorded.
Check yourself
Lumo: 3 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.
Lumo: ready for the next step? Take your time: you can come back to this page whenever you need it.
Lumo: nicely done! One more page on your journey.