Your learning journey
New to Kolla-Ansible Day-2 operations? Follow these five stages in order. Each stage groups a few pages of this study,
from the easiest to the most detailed. You do not need to read everything at once: come back whenever you like.
Lumo, your guide
Here is the map of the island: five stages, from the harbour to the observatory. Go at your own pace. When the
source code does not establish something, the page says NOT AVAILABLE : we never guess.
1 Discover2 Understand3 Practise4 Judge the risks5 Master
How to use the journey
Open a page of the stage you are at, read it, then tick Mark this page as read at the bottom.
The bar under the title of each page tells you where you are, and the buttons take you to the previous or next
step. Many pages end with To remember , the few ideas to keep, then What would you do? , a situation from a real platform with three ways to
react, and Check yourself , a short quiz. Each stage ends with a final challenge
below: read its pages and pass the challenge to earn the seal of the stage. Your progress is kept in this browser
only: nothing is sent anywhere.
Why this order
The first stages explain how the pages are written and what happens when a command runs. With that in mind,
the procedures of the later stages make sense instead of being recipes to copy.
Lumo: practise safely
The study is a reading of the source code: nothing in it was run against a host. Try what you learn on a
test platform, never first on production.
Three levels
Each page of this study shows its level under its title, so that you know what to expect:
Discover the conventions, the essentials and the concepts: start here.
Practise the operations, the risks and troubleshooting: once the concepts are clear.
Master the components in depth and the detailed evidence: to go further.
The five stages
Stage 1 of 5 · completed
Discover
Get your bearings: how the pages are written, the essentials, and the words used everywhere.
Conventions and evidence labels read
Ten things to know before a production run read
Glossary read
Find it fast read
Final challenge of the stage: 5 questions
Lumo: 5 questions on the whole stage. Get at least 4
right on the first try, with every page of the stage marked as read, and the seal of
Discover is yours. You can try again as often as you like.
1. A cloud deployed by 2023.1 never set om_enable_rabbitmq_quorum_queues. Why does its first 2024.1 run restart the messaging services, computes included, with no operator change?
Because 2024.1 recreates every container to move RabbitMQ to a new image
Because the default is now true, which writes rabbit_quorum_queue = true into the service configs: a changed file, so a restart
Because prechecks converts the classic queues and then restarts the services
Show the answer
Answer: Because the default is now true, which writes rabbit_quorum_queue = true into the service configs: a changed file, so a restart. om_enable_rabbitmq_quorum_queues now defaults to true and writes rabbit_quorum_queue = true into 26 templates of 24 roles, nova-cell's nova.conf included, so those services restart CODE . prechecks changes nothing: it fails while classic queues exist. Settle the queue type before that run.
Read the page .
2. Where do the notified restart handlers of most roles run, and which role deliberately waits?
At the meta: flush_handlers at the end of the role's deploy.yml (or else at the end of the play); the loadbalancer role defers to the end of its play
Right after each task that notifies them, in every role
Only at the very end of the whole site.yml run, after the last play
Show the answer
Answer: At the meta: flush_handlers at the end of the role's deploy.yml (or else at the end of the play); the loadbalancer role defers to the end of its play. meta: flush_handlers is where the notified restart handlers run: at the end of most deploy.yml (e.g. nova's), or else at the end of the play. The loadbalancer role (haproxy, keepalived, proxysql) deliberately defers to the end of its play CODE .
Read the page .
3. Which command is classed read-only on the operation pages, and on what is that based?
genconfig, because it restarts no container
pull, because running containers do not change
prechecks: the task files it reaches only read state (wait_for, assert, stat, container facts, and commands such as the new RabbitMQ queue listing)
Show the answer
Answer: prechecks: the task files it reaches only read state (wait_for, assert, stat, container facts, and commands such as the new RabbitMQ queue listing). read-only is supported only by prechecks (kolla_action=precheck): its task files use only checks and commands that read state, such as {{ kolla_container_engine }} --version or the new RabbitMQ queue listing CODE . genconfig is writes-config and pull is pulls-images.
Read the page .
4. A service uses config_strategy COPY_ONCE instead of the default COPY_ALWAYS. What does the restart handler do when its config has changed?
It restarts the systemd unit, and kolla_set_configs copies the new config at that start
It always recreates the container: under COPY_ONCE the config is copied only at the first start, so only a recreate re-applies it
It sends SIGHUP so that the service reloads its files
Show the answer
Answer: It always recreates the container: under COPY_ONCE the config is copied only at the first start, so only a recreate re-applies it. Under COPY_ONCE, kolla_set_configs copies the config only at the first start of a container, and the restart handler then always recreates the container, with Docker or Podman, so only a recreate re-applies config CODE .
Read the page .
5. During an incident, what can the portal tell you, and what mostly not?
What Kolla will do given the inputs (tasks and restarts); what that does to VMs and APIs depends on the environment and is mostly NOT AVAILABLE
What happened to your VMs during the last run, because the study observed real runs
Nothing: the study only covers initial deployment, not incidents
Show the answer
Answer: What Kolla will do given the inputs (tasks and restarts); what that does to VMs and APIs depends on the environment and is mostly NOT AVAILABLE . The portal can tell what Kolla will do given the inputs: tasks and restarts. What that does to VMs and APIs depends on the environment and is mostly NOT AVAILABLE in these sources: no run was used as evidence.
Read the page .
Stage 2 of 5 · completed
Understand
See how Kolla-Ansible is built and what happens, step by step, when a command runs.
Repository architecture read
Deployment execution flow read
Kolla images: start path and one-shot containers read
Compute scaling and performance read
Final challenge of the stage: 5 questions
Lumo: 5 questions on the whole stage. Get at least 4
right on the first try, with every page of the stage marked as read, and the seal of
Understand is yours. You can try again as often as you like.
1. You set kolla_max_fail_percentage: 30 and run reconfigure with ANSIBLE_SERIAL=4. One host fails in each of the first three batches. Does the play stop?
Yes: three failed hosts against a batch of four is 75%, above 30%
No: ansible-core compares the hosts failed in the current batch with the batch size (1 of 4, 25%, each time); the failed list is cleared at each batch
No: max_fail_percentage is ignored whenever serial is set
Show the answer
Answer: No: ansible-core compares the hosts failed in the current batch with the batch size (1 of 4, 25%, each time); the failed list is cleared at each batch. New in 2024.1, every play sets max_fail_percentage (default 100, which never stops a play early). ansible-core 2.16.20 compares the hosts failed in the current batch with the batch size and clears the failed-host list at the start of each batch, so failures do not add up across batches CODE .
Read the page .
2. With enable_etcd, an etcd config change is applied by reconfigure. Which member do the 2024.1 handlers restart last?
The leader: the handlers look up the leader, restart the non-leaders, then the leader, whose handler is defined last
The first host of the etcd group, because run_once targets it
The last host of the inventory, because batches follow inventory order
Show the answer
Answer: The leader: the handlers look up the leader, restart the non-leaders, then the leader, whose handler is defined last. Restart etcd container is now only a name that handlers listen to: they look up the leader, restart the non-leaders, then the leader, which comes last because its handler is defined last; each restart waits for the client port and etcdctl endpoint health CODE . Whether the % 4 groups of non-leaders really restart one after another is NOT AVAILABLE .
Read the page .
3. In 2024.1, what does the systemd unit kolla-<name>-container.service of a container say about the engine?
Nothing: it always calls docker, and Podman containers have no unit
It names the engine: ExecStart and ExecStop call /usr/bin/<engine>, and After/Wants is docker.service for Docker, network-online.target for Podman
It only holds the restart policy; start and stop go through the engine API
Show the answer
Answer: It names the engine: ExecStart and ExecStop call /usr/bin/<engine>, and After/Wants is docker.service for Docker, network-online.target for Podman. Changed in 2024.1: the unit template names its engine and writes StartLimitInterval= where 2023.1 wrote StartLimitIntervalSec=; with Docker the engine lines read as the 2023.1 ones (that template hard-coded docker), so the start-limit line is what differs. Because compare_container compares the whole unit text, a unit written by 2023.1 is reported as changed, and the handler restarts the container after rewriting the unit (F16) CODE .
Read the page .
4. You change only the command in a service's config.json. Later its container restarts for another reason, under COPY_ALWAYS. Is the new command used?
No: the command is fixed when the container is created
Only if kolla_set_configs --check reported the change first
Yes: every start re-copies the listed entries and rewrites /run_command, so a changed command applies at that restart
Show the answer
Answer: Yes: every start re-copies the listed entries and rewrites /run_command, so a changed command applies at that restart. A COPY_ALWAYS restart re-copies the listed entries and rewrites /run_command: changed or added files, mode and owner changes and a changed command apply CODE . The check itself never compares command, so only a restart for another reason applies it.
Read the page .
5. Why does a no-change -t nova reconfigure still cost a lot of work on a large fleet?
Because Kolla has no "only if changed" short-circuit at play level: every compute runs the full per-host chain
Because every task restarts the nova containers to be safe
Because the image of every compute is pulled again
Show the answer
Answer: Because Kolla has no "only if changed" short-circuit at play level: every compute runs the full per-host chain. Still holds in 2024.1: every run that includes the compute group executes the full per-host chain on every compute, even with no change: config re-rendering and comparison, container config checks, sysctl and modules, discovery CODE . Nothing restarts, and the work is still paid on every host.
Read the page .
Stage 3 of 5 · completed
Practise
Prepare a command before running it: check, predict, then apply one service or one batch at a time.
Pre-execution checklist read
Predict and inspect a run read
Reconfigure a single service read
Write config first, apply in controlled batches read
Command quick lookup by tag read
Final challenge of the stage: 5 questions
Lumo: 5 questions on the whole stage. Get at least 4
right on the first try, with every page of the stage marked as read, and the seal of
Practise is yours. You can try again as often as you like.
1. Before the first 2024.1 run on a cloud deployed by 2023.1, why run kolla-ansible prechecks without -t?
Because the HA/quorum check is in the untagged prechecks play, and the queue-type check in the rabbitmq precheck, which runs only under the rabbitmq and rabbitmq-ha-precheck tags: a run tagged with other services skips both
Because a tagged prechecks restarts the tagged services
Because -t makes prechecks migrate the queues
Show the answer
Answer: Because the HA/quorum check is in the untagged prechecks play, and the queue-type check in the rabbitmq precheck, which runs only under the rabbitmq and rabbitmq-ha-precheck tags: a run tagged with other services skips both. The check that fails when om_enable_rabbitmq_high_availability and om_enable_rabbitmq_quorum_queues are equal is in the prechecks role, an untagged play that any tagged run skips (roles/prechecks/tasks/service_checks.yml:67-72). The assertion that every queue except reply_ and _fanout ones is a quorum queue is in roles/rabbitmq/tasks/precheck.yml, under the rabbitmq and rabbitmq-ha-precheck tags, so a run tagged with other services skips it too CODE .
Read the page .
2. Before the first 2024.1 run, how can you list on a host the containers that will restart because of the changed systemd unit text (F16)?
kolla_set_configs --check in each container: exit 1 marks them
grep -l StartLimitIntervalSec /etc/systemd/system/kolla-*-container.service: the 2024.1 template has no such key, so each listed unit differs from it
kolla-ansible genconfig, then a diff of /etc/kolla/
Show the answer
Answer: grep -l StartLimitIntervalSec /etc/systemd/system/kolla-*-container.service: the 2024.1 template has no such key, so each listed unit differs from it. The 2024.1 template writes StartLimitInterval= and compares the whole unit file, so each unit still holding StartLimitIntervalSec= is drift CODE . kolla_set_configs --check does not show it, and the unit is not under /etc/kolla/.
Read the page .
3. You run ANSIBLE_SERIAL=1 kolla-ansible -i multinode -t nova reconfigure with the three multinode controllers. How many times do nova_api_bootstrap (API DB sync) and nova_cell_bootstrap (cell DB sync) run?
Once each: both are run_once
Three times each: every controller runs both
3 and 1: the API DB sync is run_once in a controller play batched one host at a time; the cell DB sync is gated on the first nova-conductor
Show the answer
Answer: 3 and 1: the API DB sync is run_once in a controller play batched one host at a time; the cell DB sync is gated on the first nova-conductor. A run_once task runs once per batch of its own play, so with kolla_serial=1 the API DB sync runs once per controller. The cell DB sync is not run_once: it is gated on the first nova-conductor and runs once at any serial value CODE .
Read the page .
4. With haproxy_single_external_frontend: true (new in 2024.1, default false), what does -t keystone reconfigure write on the loadbalancer hosts besides keystone's services.d file?
Keystone's block of external-frontend-map, which also notifies the HAProxy restart
Nothing more: the single frontend is written only by -t haproxy
The HAProxy configuration of every service
Show the answer
Answer: Keystone's block of external-frontend-map, which also notifies the HAProxy restart. The service's loadbalancer include imports loadbalancer-config with tags: always; new in 2024.1, with haproxy_single_external_frontend the service's block of external-frontend-map is also written and also notifies the restart CODE . --skip-tags haproxy drops the whole loadbalancer play.
Read the page .
5. You run -t rabbitmq reconfigure on a cloud whose RabbitMQ is already running. Which one-shot container can it start?
rabbitmq_bootstrap, on every RabbitMQ host
None: RabbitMQ has no one-shot containers
rabbitmq_version_check, on the first RabbitMQ host when the container exists; rabbitmq_bootstrap only for a new volume
Show the answer
Answer: rabbitmq_version_check, on the first RabbitMQ host when the container exists; rabbitmq_bootstrap only for a new volume. The quick lookup row for -t rabbitmq reconfigure: rabbitmq_version_check runs on the first RabbitMQ host when the container exists, and rabbitmq_bootstrap only for a new volume; the brokers restart one by one with serial: "33%", each node drained first CODE .
Read the page .
Stage 4 of 5 · completed
Judge the risks
Learn what a tag really touches, why small changes can grow, and which risks and bugs are known.
What a tag really executes read
Why a small change creates a large workload read
Risk register read
Known bugs and limitations read
Final challenge of the stage: 5 questions
Lumo: 5 questions on the whole stage. Get at least 4
right on the first try, with every page of the stage marked as read, and the seal of
Judge the risks is yours. You can try again as often as you like.
1. You set a new enable_* flag in globals.yml and run reconfigure -t <svc> for that service only. What happens to common's cron logrotate config?
It is updated on every host, because every service play runs common
It is updated on the hosts of that service only
It stays stale until -t common or an untagged run
Show the answer
Answer: It stays stale until -t common or an untagged run. Common's cron logrotate config lists services by their enable_* flags CODE . Still holds in 2024.1: only the common play renders it, and that play is tagged common only, so enabling a service with -t <svc> leaves it stale until -t common or an untagged run.
Read the page .
2. You want one nova-compute option on one compute only. Which override file reaches that compute only in 2024.1?
/etc/kolla/config/nova/nova-compute.conf
/etc/kolla/config/nova/<host>/nova-compute.conf, new in 2024.1, merged into that host's nova-compute/nova.conf only
/etc/kolla/config/global.conf, which is read per host
Show the answer
Answer: /etc/kolla/config/nova/<host>/nova-compute.conf, new in 2024.1, merged into that host's nova-compute/nova.conf only. nova/nova-compute.conf targets every compute. New in 2024.1, the same option in nova/<host>/nova-compute.conf is merged into that host's nova-compute/nova.conf only CODE .
Read the page .
3. You want to keep classic RabbitMQ queues on 2024.1. Which setting is the only one that passes prechecks?
om_enable_rabbitmq_high_availability: true with quorum queues off and the ha-all policy in place
Both flags set to false, the 2023.1 defaults
Both flags set to true
Show the answer
Answer: om_enable_rabbitmq_high_availability: true with quorum queues off and the ha-all policy in place. This is R27 CODE : prechecks fails when HA and quorum are equal (both off or both on), and with quorum on it asserts that the queues are quorum queues. The only classic-queue setting that passes is HA on, quorum off, with the ha-all policy in place; on a cloud that kept the 2023.1 default (HA off), 2023.1 runs removed that policy, so this path too is a manual migration, to mirrored queues, as the HA precheck message says. What oslo.messaging does with existing classic queues is NOT AVAILABLE .
Read the page .
4. A cloud has enable_octavia: yes, the default amphora provider and enable_redis left at no. What does 2024.1 do?
It deploys Redis automatically for the Octavia jobboard
Nothing changes: the jobboard is off by default
prechecks fails ("Redis must be enabled when using octavia jobboard"); deploy, reconfigure and upgrade do not run that assert and render the jobboard section anyway
Show the answer
Answer: prechecks fails ("Redis must be enabled when using octavia jobboard"); deploy, reconfigure and upgrade do not run that assert and render the jobboard section anyway. This is R30 CODE : enable_octavia_jobboard is on by default with the amphora provider, so octavia.conf lists the redis group as jobboard hosts. How Octavia behaves when they hold no Redis is NOT AVAILABLE . Choose enable_redis: yes or enable_octavia_jobboard: no before the upgrade.
Read the page .
5. You turn haproxy_single_external_frontend on, then off again. What happens to services.d/external-frontend.cfg?
It is removed at the next -t haproxy reconfigure
It stays in place: services.d files are never removed (the only removal is mariadb.cfg when ProxySQL is enabled)
It is removed by genconfig
Show the answer
Answer: It stays in place: services.d files are never removed (the only removal is mariadb.cfg when ProxySQL is enabled). Stale services.d files are never removed, and new in 2024.1, turning haproxy_single_external_frontend off leaves services.d/external-frontend.cfg in place; a disabled service's block also stays in external-frontend-map CODE .
Read the page .
Stage 5 of 5 · completed
Master
Go deeper: how the components depend on each other, the detailed findings, and the limits of what is known.
Component dependencies read
Nova deep dive read
Technical findings read
Method and limits of the study read
What is not known read
Final challenge of the stage: 5 questions
Lumo: 5 questions on the whole stage. Get at least 4
right on the first try, with every page of the stage marked as read, and the seal of
Master is yours. You can try again as often as you like.
1. On a fresh cloud where Keystone was never deployed, you run kolla-ansible deploy -t nova. What happens?
Kolla deploys Keystone first, because the nova role declares a dependency on it
Nothing enforces a dependency when you use tags: the run reaches Keystone registration with no Keystone deployed
Nova waits until Keystone is deployed by a later run, then registers itself
Show the answer
Answer: Nothing enforces a dependency when you use tags: the run reaches Keystone registration with no Keystone deployed. --tags filters plays, and no role has a meta directory in 2024.1, so -t nova on a fresh cloud reaches Keystone registration (service-ks-register, against keystone_internal_url) with no Keystone deployed CODE .
Read the page .
2. During an upgrade with nova_safety_upgrade: yes, which nova containers keep running?
nova_compute, nova_compute_ironic, nova_libvirt and nova_ssh; the API, scheduler, conductors and console proxies stay down until the reload plays start them again
None: every nova container is stopped first
Only the API services, so that users see no outage
Show the answer
Answer: nova_compute, nova_compute_ironic, nova_libvirt and nova_ssh; the API, scheduler, conductors and console proxies stay down until the reload plays start them again. nova_safety_upgrade stops nova_api, nova_scheduler, nova_super_conductor and, on the cell hosts, the conductor and console proxies before the upgrade; the compute-side containers keep running and their restart handlers are not skipped CODE .
Read the page .
3. You run kolla-ansible deploy with --limit set to one new compute host. Why does work still run on control01?
Because --limit is ignored by every play that includes the compute group
It does not: --limit confines every action to the hosts you name
Because delegate_to: groups[x][0] runs work on hosts outside the limit: the Neutron DB create/grant, the bootstrap_neutron container and cell discovery
Show the answer
Answer: Because delegate_to: groups[x][0] runs work on hosts outside the limit: the Neutron DB create/grant, the bootstrap_neutron container and cell discovery. Still holds in 2024.1 (F8): --limit does not confine actions CODE . With a compute host as the limit, control01 runs the Neutron DB create/grant, the bootstrap_neutron container and cell discovery, so "only the new compute" still executes DB-sync containers on a controller.
Read the page .
4. Kolla-Ansible 2024.1 also accepts ansible-core 2.15. If you run it with 2.15, do the study's statements about Ansible behaviour (loop-notify, run_once with serial, max_fail_percentage) still hold?
Yes: the study validated them with lab runs on both versions
Not established: only ansible-core 2.16.20 was read, so the behaviour of 2.15 is not available
No: the study shows that 2.15 has no loop-notify
Show the answer
Answer: Not established: only ansible-core 2.16.20 was read, so the behaviour of 2.15 is not available. Only one ansible-core version was read: Kolla-Ansible 2024.1 accepts 2.15 to 2.16, and the behaviour of 2.15 is NOT AVAILABLE . The study rests only on reading source code, with no runtime or lab evidence.
Read the page .
5. New in 2024.1, the openvswitch post-config also sets external_ids:hostname, and a change of it notifies the openvswitch_vswitchd restart. Does the first 2024.1 run restart openvswitch_vswitchd on every compute?
Yes, always, because the key is new
No, never, because post-config runs only on new hosts
Not established: whether the run reports the new key as changed depends on the openvswitch_db module, whose code is not in the studied sources
Show the answer
Answer: Not established: whether the run reports the new key as changed depends on the openvswitch_db module, whose code is not in the studied sources. The notify is CODE , but whether the first 2024.1 run reports external_ids:hostname changed, and so restarts openvswitch_vswitchd, is NOT AVAILABLE : the openvswitch_db module is not in these sources.
Read the page .
beginner
learning