Skip to content

Your learning journey

New to Kolla-Ansible Day-2 operations? Follow these five stages in order. Each stage groups a few pages of this study, from the easiest to the most detailed. You do not need to read everything at once: come back whenever you like.

Lumo, the guide, holds open a parchment map.

Lumo, your guide

Here is the map of the island: five stages, from the harbour to the observatory. Go at your own pace. When the source code does not establish something, the page says NOT AVAILABLE: we never guess.

A winding dotted path connects a harbour gate, library, workshop, lighthouse and cliffside observatory across an island.
1 Discover2 Understand3 Practise4 Judge the risks5 Master

How to use the journey

Open a page of the stage you are at, read it, then tick Mark this page as read at the bottom. The bar under the title of each page tells you where you are, and the buttons take you to the previous or next step. Many pages end with To remember, the few ideas to keep, then What would you do?, a situation from a real platform with three ways to react, and Check yourself, a short quiz. Each stage ends with a final challenge below: read its pages and pass the challenge to earn the seal of the stage. Your progress is kept in this browser only: nothing is sent anywhere.

Why this order

The first stages explain how the pages are written and what happens when a command runs. With that in mind, the procedures of the later stages make sense instead of being recipes to copy.

Lumo wears brass goggles while adjusting a small brass server machine.

Lumo: practise safely

The study is a reading of the source code: nothing in it was run against a host. Try what you learn on a test platform, never first on production.

Three levels

Each page of this study shows its level under its title, so that you know what to expect:

  • Discover the conventions, the essentials and the concepts: start here.
  • Practise the operations, the risks and troubleshooting: once the concepts are clear.
  • Master the components in depth and the detailed evidence: to go further.

The five stages

  1. A stone harbour gate stands beside a docked boat, wooden crates and turquoise water.

    Stage 1 of 5

    Discover

    Get your bearings: how the pages are written, the essentials, and the words used everywhere.

    1. Conventions and evidence labels
    2. Ten things to know before a production run
    3. Glossary
    4. Find it fast
    Final challenge of the stage: 5 questions

    Lumo: 5 questions on the whole stage. Get at least 4 right on the first try, with every page of the stage marked as read, and the seal of Discover is yours. You can try again as often as you like.

    1. A cloud deployed by 2023.1 never set om_enable_rabbitmq_quorum_queues. Why does its first 2024.1 run restart the messaging services, computes included, with no operator change?
    Show the answer

    Answer: Because the default is now true, which writes rabbit_quorum_queue = true into the service configs: a changed file, so a restart. om_enable_rabbitmq_quorum_queues now defaults to true and writes rabbit_quorum_queue = true into 26 templates of 24 roles, nova-cell's nova.conf included, so those services restart CODE. prechecks changes nothing: it fails while classic queues exist. Settle the queue type before that run. Read the page.

    2. Where do the notified restart handlers of most roles run, and which role deliberately waits?
    Show the answer

    Answer: At the meta: flush_handlers at the end of the role's deploy.yml (or else at the end of the play); the loadbalancer role defers to the end of its play. meta: flush_handlers is where the notified restart handlers run: at the end of most deploy.yml (e.g. nova's), or else at the end of the play. The loadbalancer role (haproxy, keepalived, proxysql) deliberately defers to the end of its play CODE. Read the page.

    3. Which command is classed read-only on the operation pages, and on what is that based?
    Show the answer

    Answer: prechecks: the task files it reaches only read state (wait_for, assert, stat, container facts, and commands such as the new RabbitMQ queue listing). read-only is supported only by prechecks (kolla_action=precheck): its task files use only checks and commands that read state, such as {{ kolla_container_engine }} --version or the new RabbitMQ queue listing CODE. genconfig is writes-config and pull is pulls-images. Read the page.

    4. A service uses config_strategy COPY_ONCE instead of the default COPY_ALWAYS. What does the restart handler do when its config has changed?
    Show the answer

    Answer: It always recreates the container: under COPY_ONCE the config is copied only at the first start, so only a recreate re-applies it. Under COPY_ONCE, kolla_set_configs copies the config only at the first start of a container, and the restart handler then always recreates the container, with Docker or Podman, so only a recreate re-applies config CODE. Read the page.

    5. During an incident, what can the portal tell you, and what mostly not?
    Show the answer

    Answer: What Kolla will do given the inputs (tasks and restarts); what that does to VMs and APIs depends on the environment and is mostly NOT AVAILABLE. The portal can tell what Kolla will do given the inputs: tasks and restarts. What that does to VMs and APIs depends on the environment and is mostly NOT AVAILABLE in these sources: no run was used as evidence. Read the page.

  2. A white stone library tower opens its arched windows above the island sea.

    Stage 2 of 5

    Understand

    See how Kolla-Ansible is built and what happens, step by step, when a command runs.

    1. Repository architecture
    2. Deployment execution flow
    3. Kolla images: start path and one-shot containers
    4. Compute scaling and performance
    Final challenge of the stage: 5 questions

    Lumo: 5 questions on the whole stage. Get at least 4 right on the first try, with every page of the stage marked as read, and the seal of Understand is yours. You can try again as often as you like.

    1. You set kolla_max_fail_percentage: 30 and run reconfigure with ANSIBLE_SERIAL=4. One host fails in each of the first three batches. Does the play stop?
    Show the answer

    Answer: No: ansible-core compares the hosts failed in the current batch with the batch size (1 of 4, 25%, each time); the failed list is cleared at each batch. New in 2024.1, every play sets max_fail_percentage (default 100, which never stops a play early). ansible-core 2.16.20 compares the hosts failed in the current batch with the batch size and clears the failed-host list at the start of each batch, so failures do not add up across batches CODE. Read the page.

    2. With enable_etcd, an etcd config change is applied by reconfigure. Which member do the 2024.1 handlers restart last?
    Show the answer

    Answer: The leader: the handlers look up the leader, restart the non-leaders, then the leader, whose handler is defined last. Restart etcd container is now only a name that handlers listen to: they look up the leader, restart the non-leaders, then the leader, which comes last because its handler is defined last; each restart waits for the client port and etcdctl endpoint health CODE. Whether the % 4 groups of non-leaders really restart one after another is NOT AVAILABLE. Read the page.

    3. In 2024.1, what does the systemd unit kolla-<name>-container.service of a container say about the engine?
    Show the answer

    Answer: It names the engine: ExecStart and ExecStop call /usr/bin/<engine>, and After/Wants is docker.service for Docker, network-online.target for Podman. Changed in 2024.1: the unit template names its engine and writes StartLimitInterval= where 2023.1 wrote StartLimitIntervalSec=; with Docker the engine lines read as the 2023.1 ones (that template hard-coded docker), so the start-limit line is what differs. Because compare_container compares the whole unit text, a unit written by 2023.1 is reported as changed, and the handler restarts the container after rewriting the unit (F16) CODE. Read the page.

    4. You change only the command in a service's config.json. Later its container restarts for another reason, under COPY_ALWAYS. Is the new command used?
    Show the answer

    Answer: Yes: every start re-copies the listed entries and rewrites /run_command, so a changed command applies at that restart. A COPY_ALWAYS restart re-copies the listed entries and rewrites /run_command: changed or added files, mode and owner changes and a changed command apply CODE. The check itself never compares command, so only a restart for another reason applies it. Read the page.

    5. Why does a no-change -t nova reconfigure still cost a lot of work on a large fleet?
    Show the answer

    Answer: Because Kolla has no "only if changed" short-circuit at play level: every compute runs the full per-host chain. Still holds in 2024.1: every run that includes the compute group executes the full per-host chain on every compute, even with no change: config re-rendering and comparison, container config checks, sysctl and modules, discovery CODE. Nothing restarts, and the work is still paid on every host. Read the page.

  3. Brass machinery and server devices sit on a workbench in a bright island workshop.

    Stage 3 of 5

    Practise

    Prepare a command before running it: check, predict, then apply one service or one batch at a time.

    1. Pre-execution checklist
    2. Predict and inspect a run
    3. Reconfigure a single service
    4. Write config first, apply in controlled batches
    5. Command quick lookup by tag
    Final challenge of the stage: 5 questions

    Lumo: 5 questions on the whole stage. Get at least 4 right on the first try, with every page of the stage marked as read, and the seal of Practise is yours. You can try again as often as you like.

    1. Before the first 2024.1 run on a cloud deployed by 2023.1, why run kolla-ansible prechecks without -t?
    Show the answer

    Answer: Because the HA/quorum check is in the untagged prechecks play, and the queue-type check in the rabbitmq precheck, which runs only under the rabbitmq and rabbitmq-ha-precheck tags: a run tagged with other services skips both. The check that fails when om_enable_rabbitmq_high_availability and om_enable_rabbitmq_quorum_queues are equal is in the prechecks role, an untagged play that any tagged run skips (roles/prechecks/tasks/service_checks.yml:67-72). The assertion that every queue except reply_ and _fanout ones is a quorum queue is in roles/rabbitmq/tasks/precheck.yml, under the rabbitmq and rabbitmq-ha-precheck tags, so a run tagged with other services skips it too CODE. Read the page.

    2. Before the first 2024.1 run, how can you list on a host the containers that will restart because of the changed systemd unit text (F16)?
    Show the answer

    Answer: grep -l StartLimitIntervalSec /etc/systemd/system/kolla-*-container.service: the 2024.1 template has no such key, so each listed unit differs from it. The 2024.1 template writes StartLimitInterval= and compares the whole unit file, so each unit still holding StartLimitIntervalSec= is drift CODE. kolla_set_configs --check does not show it, and the unit is not under /etc/kolla/. Read the page.

    3. You run ANSIBLE_SERIAL=1 kolla-ansible -i multinode -t nova reconfigure with the three multinode controllers. How many times do nova_api_bootstrap (API DB sync) and nova_cell_bootstrap (cell DB sync) run?
    Show the answer

    Answer: 3 and 1: the API DB sync is run_once in a controller play batched one host at a time; the cell DB sync is gated on the first nova-conductor. A run_once task runs once per batch of its own play, so with kolla_serial=1 the API DB sync runs once per controller. The cell DB sync is not run_once: it is gated on the first nova-conductor and runs once at any serial value CODE. Read the page.

    4. With haproxy_single_external_frontend: true (new in 2024.1, default false), what does -t keystone reconfigure write on the loadbalancer hosts besides keystone's services.d file?
    Show the answer

    Answer: Keystone's block of external-frontend-map, which also notifies the HAProxy restart. The service's loadbalancer include imports loadbalancer-config with tags: always; new in 2024.1, with haproxy_single_external_frontend the service's block of external-frontend-map is also written and also notifies the restart CODE. --skip-tags haproxy drops the whole loadbalancer play. Read the page.

    5. You run -t rabbitmq reconfigure on a cloud whose RabbitMQ is already running. Which one-shot container can it start?
    Show the answer

    Answer: rabbitmq_version_check, on the first RabbitMQ host when the container exists; rabbitmq_bootstrap only for a new volume. The quick lookup row for -t rabbitmq reconfigure: rabbitmq_version_check runs on the first RabbitMQ host when the container exists, and rabbitmq_bootstrap only for a new volume; the brokers restart one by one with serial: "33%", each node drained first CODE. Read the page.

  4. A white stone lighthouse watches over a calm rocky coast and a small sailing ship.

    Stage 4 of 5

    Judge the risks

    Learn what a tag really touches, why small changes can grow, and which risks and bugs are known.

    1. What a tag really executes
    2. Why a small change creates a large workload
    3. Risk register
    4. Known bugs and limitations
    Final challenge of the stage: 5 questions

    Lumo: 5 questions on the whole stage. Get at least 4 right on the first try, with every page of the stage marked as read, and the seal of Judge the risks is yours. You can try again as often as you like.

    1. You set a new enable_* flag in globals.yml and run reconfigure -t <svc> for that service only. What happens to common's cron logrotate config?
    Show the answer

    Answer: It stays stale until -t common or an untagged run. Common's cron logrotate config lists services by their enable_* flags CODE. Still holds in 2024.1: only the common play renders it, and that play is tagged common only, so enabling a service with -t <svc> leaves it stale until -t common or an untagged run. Read the page.

    2. You want one nova-compute option on one compute only. Which override file reaches that compute only in 2024.1?
    Show the answer

    Answer: /etc/kolla/config/nova/<host>/nova-compute.conf, new in 2024.1, merged into that host's nova-compute/nova.conf only. nova/nova-compute.conf targets every compute. New in 2024.1, the same option in nova/<host>/nova-compute.conf is merged into that host's nova-compute/nova.conf only CODE. Read the page.

    3. You want to keep classic RabbitMQ queues on 2024.1. Which setting is the only one that passes prechecks?
    Show the answer

    Answer: om_enable_rabbitmq_high_availability: true with quorum queues off and the ha-all policy in place. This is R27 CODE: prechecks fails when HA and quorum are equal (both off or both on), and with quorum on it asserts that the queues are quorum queues. The only classic-queue setting that passes is HA on, quorum off, with the ha-all policy in place; on a cloud that kept the 2023.1 default (HA off), 2023.1 runs removed that policy, so this path too is a manual migration, to mirrored queues, as the HA precheck message says. What oslo.messaging does with existing classic queues is NOT AVAILABLE. Read the page.

    4. A cloud has enable_octavia: yes, the default amphora provider and enable_redis left at no. What does 2024.1 do?
    Show the answer

    Answer: prechecks fails ("Redis must be enabled when using octavia jobboard"); deploy, reconfigure and upgrade do not run that assert and render the jobboard section anyway. This is R30 CODE: enable_octavia_jobboard is on by default with the amphora provider, so octavia.conf lists the redis group as jobboard hosts. How Octavia behaves when they hold no Redis is NOT AVAILABLE. Choose enable_redis: yes or enable_octavia_jobboard: no before the upgrade. Read the page.

    5. You turn haproxy_single_external_frontend on, then off again. What happens to services.d/external-frontend.cfg?
    Show the answer

    Answer: It stays in place: services.d files are never removed (the only removal is mariadb.cfg when ProxySQL is enabled). Stale services.d files are never removed, and new in 2024.1, turning haproxy_single_external_frontend off leaves services.d/external-frontend.cfg in place; a disabled service's block also stays in external-frontend-map CODE. Read the page.

  5. A domed observatory with a brass telescope overlooks the sea from a high cliff.

    Stage 5 of 5

    Master

    Go deeper: how the components depend on each other, the detailed findings, and the limits of what is known.

    1. Component dependencies
    2. Nova deep dive
    3. Technical findings
    4. Method and limits of the study
    5. What is not known
    Final challenge of the stage: 5 questions

    Lumo: 5 questions on the whole stage. Get at least 4 right on the first try, with every page of the stage marked as read, and the seal of Master is yours. You can try again as often as you like.

    1. On a fresh cloud where Keystone was never deployed, you run kolla-ansible deploy -t nova. What happens?
    Show the answer

    Answer: Nothing enforces a dependency when you use tags: the run reaches Keystone registration with no Keystone deployed. --tags filters plays, and no role has a meta directory in 2024.1, so -t nova on a fresh cloud reaches Keystone registration (service-ks-register, against keystone_internal_url) with no Keystone deployed CODE. Read the page.

    2. During an upgrade with nova_safety_upgrade: yes, which nova containers keep running?
    Show the answer

    Answer: nova_compute, nova_compute_ironic, nova_libvirt and nova_ssh; the API, scheduler, conductors and console proxies stay down until the reload plays start them again. nova_safety_upgrade stops nova_api, nova_scheduler, nova_super_conductor and, on the cell hosts, the conductor and console proxies before the upgrade; the compute-side containers keep running and their restart handlers are not skipped CODE. Read the page.

    3. You run kolla-ansible deploy with --limit set to one new compute host. Why does work still run on control01?
    Show the answer

    Answer: Because delegate_to: groups[x][0] runs work on hosts outside the limit: the Neutron DB create/grant, the bootstrap_neutron container and cell discovery. Still holds in 2024.1 (F8): --limit does not confine actions CODE. With a compute host as the limit, control01 runs the Neutron DB create/grant, the bootstrap_neutron container and cell discovery, so "only the new compute" still executes DB-sync containers on a controller. Read the page.

    4. Kolla-Ansible 2024.1 also accepts ansible-core 2.15. If you run it with 2.15, do the study's statements about Ansible behaviour (loop-notify, run_once with serial, max_fail_percentage) still hold?
    Show the answer

    Answer: Not established: only ansible-core 2.16.20 was read, so the behaviour of 2.15 is not available. Only one ansible-core version was read: Kolla-Ansible 2024.1 accepts 2.15 to 2.16, and the behaviour of 2.15 is NOT AVAILABLE. The study rests only on reading source code, with no runtime or lab evidence. Read the page.

    5. New in 2024.1, the openvswitch post-config also sets external_ids:hostname, and a change of it notifies the openvswitch_vswitchd restart. Does the first 2024.1 run restart openvswitch_vswitchd on every compute?
    Show the answer

    Answer: Not established: whether the run reports the new key as changed depends on the openvswitch_db module, whose code is not in the studied sources. The notify is CODE, but whether the first 2024.1 run reports external_ids:hostname changed, and so restarts openvswitch_vswitchd, is NOT AVAILABLE: the openvswitch_db module is not in these sources. Read the page.