Skip to content

Your learning journey

New to Kolla-Ansible Day-2 operations? Follow these five stages in order. Each stage groups a few pages of this study, from the easiest to the most detailed. You do not need to read everything at once: come back whenever you like.

Lumo, the guide, holds open a parchment map.

Lumo, your guide

Here is the map of the island: five stages, from the harbour to the observatory. Go at your own pace. When the source code does not establish something, the page says NOT AVAILABLE: we never guess.

A winding dotted path connects a harbour gate, library, workshop, lighthouse and cliffside observatory across an island.
1 Discover2 Understand3 Practise4 Judge the risks5 Master

How to use the journey

Open a page of the stage you are at, read it, then tick Mark this page as read at the bottom. The bar under the title of each page tells you where you are, and the buttons take you to the previous or next step. Many pages end with To remember, the few ideas to keep, then What would you do?, a situation from a real platform with three ways to react, and Check yourself, a short quiz. Each stage ends with a final challenge below: read its pages and pass the challenge to earn the seal of the stage. Your progress is kept in this browser only: nothing is sent anywhere.

Why this order

The first stages explain how the pages are written and what happens when a command runs. With that in mind, the procedures of the later stages make sense instead of being recipes to copy.

Lumo wears brass goggles while adjusting a small brass server machine.

Lumo: practise safely

The study is a reading of the source code: nothing in it was run against a host. Try what you learn on a test platform, never first on production.

Three levels

Each page of this study shows its level under its title, so that you know what to expect:

  • Discover the conventions, the essentials and the concepts: start here.
  • Practise the operations, the risks and troubleshooting: once the concepts are clear.
  • Master the components in depth and the detailed evidence: to go further.

The five stages

  1. A stone harbour gate stands beside a docked boat, wooden crates and turquoise water.

    Stage 1 of 5

    Discover

    Get your bearings: how the pages are written, the essentials, and the words used everywhere.

    1. Conventions and evidence labels
    2. Ten things to know before a production run
    3. Glossary
    4. Find it fast
    Final challenge of the stage: 5 questions

    Lumo: 5 questions on the whole stage. Get at least 4 right on the first try, with every page of the stage marked as read, and the seal of Discover is yours. You can try again as often as you like.

    1. You change one global setting on the deployment host, for example a RabbitMQ or memcached membership or a password. Why can this small central change restart containers across the whole fleet?
    Show the answer

    Answer: Because global settings land in per-host files, including compute-side configs. Global settings land in per-host files: enable_*, RabbitMQ/memcached membership and passwords end up in compute-side configs, so small central changes cause fleet-wide restarts. Read the page.

    2. You rebuilt an image and pushed it under the same tag, then ran deploy. The containers still run the old image. Why?
    Show the answer

    Answer: deploy never pulls an image that is already present; rebuilt images under the same tag need pull. deploy never pulls an image that is already present. Rebuilt images under the same tag need pull. Read the page.

    3. A service uses config_strategy COPY_ONCE instead of the default COPY_ALWAYS. What happens to a config change when its container is only restarted?
    Show the answer

    Answer: It is not re-applied: under COPY_ONCE the config is copied only on the first start, so only a recreate re-applies it. COPY_ALWAYS, the default, copies the config and applies permissions on every container start. COPY_ONCE does it only on the first start of a container, so only a recreate re-applies config CODE. Read the page.

    4. An operation page shows the impact class restarts-on-drift under its title. What does that tell you?
    Show the answer

    Answer: At least one command of the page can restart containers, only if drift is detected. restarts-on-drift means the command can restart containers, only if drift is detected. A class says that at least one command of the page has that impact, not that all do; "renders config and restarts no Docker container" is writes-config. Read the page.

    5. During an incident, what can the portal tell you, and what mostly not?
    Show the answer

    Answer: What Kolla will do given the inputs (tasks and restarts); what that does to VMs and APIs depends on the environment and is mostly NOT AVAILABLE. During an incident, the portal can tell what Kolla will do given the inputs: tasks and restarts. What that does to VMs and APIs depends on the environment and is mostly NOT AVAILABLE in these sources. Read the page.

  2. A white stone library tower opens its arched windows above the island sea.

    Stage 2 of 5

    Understand

    See how Kolla-Ansible is built and what happens, step by step, when a command runs.

    1. Repository architecture
    2. Deployment execution flow
    3. Kolla images: start path and one-shot containers
    4. Compute scaling and performance
    Final challenge of the stage: 5 questions

    Lumo: 5 questions on the whole stage. Get at least 4 right on the first try, with every page of the stage marked as read, and the seal of Understand is yours. You can try again as often as you like.

    1. globals.yml sets a variable, and you set a different value for one host in the inventory host_vars. Which value does that host get, and why?
    Show the answer

    Answer: The globals.yml value: it is passed as extra-vars, the highest precedence in Ansible. globals.yml is passed with -e @file, which is extra-vars, so a value in globals.yml cannot be overridden per host or per group in the inventory CODE. Per-host variation must come from variables that are not set in globals.yml. Read the page.

    2. With kolla_serial set, the nova-cell play runs batch by batch over the computes. When are the neutron agents on compute001 restarted?
    Show the answer

    Answer: Only after all computes have finished the nova-cell and openvswitch plays. A batch runs the complete play, handler restarts included, before the next batch starts, but each play is still traversed batch by batch for the whole fleet before the next play starts. Read the page.

    3. All of memcached's settings are in the command of its config.json ("config_files": []). Why does kolla_set_configs --check never make a memcached setting change restart the container?
    Show the answer

    Answer: Because --check iterates only config_files, so this manifest always returns 0. --check never compares config.json's command and permissions: it iterates only config_files, so a manifest with "config_files": [], like memcached's, always returns 0. Read the page.

    4. Why does a no-change -t nova reconfigure still cost a lot of work on a large fleet?
    Show the answer

    Answer: Because Kolla has no "only if changed" short-circuit at play level: every compute runs the full per-host chain. Every run that includes the compute group executes the full per-host chain on every compute, even with no change: config re-rendering and comparison, container config checks, common role (in untagged runs), sysctl and modules, OVS post-config, discovery. Nothing restarts, and yet at 500 computes the run still costs about 13,000 remote invocations. Read the page.

    5. Why is restarting a container enough to apply a changed or added config file?
    Show the answer

    Answer: Because with COPY_ALWAYS, kolla_set_configs copies the files listed in config.json at every container start. The config.json manifest tells kolla_set_configs which files to copy from /var/lib/kolla/config_files; with config_strategy: COPY_ALWAYS the copy happens at every container start. That is why Kolla-Ansible never applies config by reloading a service. Read the page.

  3. Brass machinery and server devices sit on a workbench in a bright island workshop.

    Stage 3 of 5

    Practise

    Prepare a command before running it: check, predict, then apply one service or one batch at a time.

    1. Pre-execution checklist
    2. Predict and inspect a run
    3. Reconfigure a single service
    4. Write config first, apply in controlled batches
    5. Command quick lookup by tag
    Final challenge of the stage: 5 questions

    Lumo: 5 questions on the whole stage. Get at least 4 right on the first try, with every page of the stage marked as read, and the seal of Practise is yours. You can try again as often as you like.

    1. You change only the command of memcached in its config.json (for example its -m option), run genconfig, then kolla_set_configs --check as root in the memcached container. What happens?
    Show the answer

    Answer: It exits 0: command is not compared and memcached has an empty config_files, so the change is not seen as drift. 0 = every file listed in config_files matches its live copy; command and permissions are not compared, so memcached ("config_files": []) always returns 0 CODE. Such drift takes effect only at the next restart for another reason. Read the page.

    2. You run ANSIBLE_SERIAL=10 kolla-ansible -i multinode -t nova reconfigure. How often does compute discovery run?
    Show the answer

    Answer: On the first compute of each nova-cell batch, so once per batch that contains computes. A run_once task runs once per batch of its own play; compute discovery runs on the first compute of each nova-cell batch, so it repeats once per batch that contains computes CODE. Read the page.

    3. In 2023.1, does kolla-ansible -t keystone reconfigure also check and possibly restart the common containers (fluentd, kolla_toolbox, cron) on the keystone hosts?
    Show the answer

    Answer: No: common is its own play, tagged common only, so it runs only under -t common or a run without tags. No role has a meta/ directory; common is a separate play with tags: common, so a service tag never touches the common containers in 2023.1. Read the page.

    4. In 2023.1, does -t neutron reconfigure run the one-shot bootstrap_neutron container (neutron-db-manage upgrade head)?
    Show the answer

    Answer: Yes: it runs on every deploy and reconfigure, whatever neutron_enable_rolling_upgrade. The quick lookup table says that in 2023.1 bootstrap_neutron runs on every deploy and reconfigure, whatever neutron_enable_rolling_upgrade; the untagged reconfigure runs the db sync of every enabled service, neutron included. Read the page.

    5. Step 1 of the checklist runs kolla-ansible genconfig with the same tags and --limit before the real run. Why can it be run safely?
    Show the answer

    Answer: It writes config but restarts no Docker container: every container restart handler has when: kolla_action != "config". genconfig writes config but does not restart any Docker container; the handlers without that guard are no-ops, chained steps, or steps genconfig never notifies. The Ussuri nova-hyperv exception no longer exists in 2023.1. Then you diff /etc/kolla/<service>/. Read the page.

  4. A white stone lighthouse watches over a calm rocky coast and a small sailing ship.

    Stage 4 of 5

    Judge the risks

    Learn what a tag really touches, why small changes can grow, and which risks and bugs are known.

    1. What a tag really executes
    2. Why a small change creates a large workload
    3. Risk register
    4. Known bugs and limitations
    Final challenge of the stage: 5 questions

    Lumo: 5 questions on the whole stage. Get at least 4 right on the first try, with every page of the stage marked as read, and the seal of Judge the risks is yours. You can try again as often as you like.

    1. You add one extra controller to the inventory. Why is this small change a large workload?
    Show the answer

    Answer: It changes the transport URL in every messaging service everywhere. One extra controller changes the transport URL in every messaging service everywhere, so the change reaches far beyond the new host. Read the page.

    2. A config change concerns only one container of a role. Why can every container of that role on that host restart?
    Show the answer

    Answer: Because the loop result is changed if any item changed, and then the handler of every executed item is notified. Loop notify: if any item changed, the templated handler name of every executed item is notified, so a change that concerns one container restarts every container of that role on that host CODE. Skipped items do not notify. Read the page.

    3. You set a new enable_* flag in globals.yml and run reconfigure -t <svc> for that service only. What happens to common's cron logrotate config?
    Show the answer

    Answer: It stays stale until -t common or an untagged run. Common's cron logrotate config lists services by their enable_* flags CODE. Since only the common play renders it, enabling a service with -t <svc> leaves the logrotate config stale until -t common or an untagged run. Read the page.

    4. A service image is rebuilt and pushed under the same tag. You run reconfigure without pull. What happens?
    Show the answer

    Answer: Nothing changes: only absent images are pulled, and the comparison uses the local image of that tag. This is R13 CODE: nothing changes (only absent images are pulled; the comparison uses the local image of that tag). After pull: recreate, plus loop-notify restarts. Precautions: use immutable tags; always pull deliberately. Read the page.

    5. You change only the command in a container's config.json, then run genconfig + deploy-containers. Is the change applied?
    Show the answer

    Answer: No: kolla_set_configs --check compares only config_files entries, so the change is not applied until the container restarts for another reason. kolla_set_configs --check compares only config_files entries: changes confined to config.json (command, permissions) are never detected, so neither genconfig + deploy-containers nor a later run applies them until the container restarts for another reason CODE. Read the page.

  5. A domed observatory with a brass telescope overlooks the sea from a high cliff.

    Stage 5 of 5

    Master

    Go deeper: how the components depend on each other, the detailed findings, and the limits of what is known.

    1. Component dependencies
    2. Nova deep dive
    3. Technical findings
    4. Method and limits of the study
    5. What is not known
    Final challenge of the stage: 5 questions

    Lumo: 5 questions on the whole stage. Get at least 4 right on the first try, with every page of the stage marked as read, and the seal of Master is yours. You can try again as often as you like.

    1. On a fresh cloud where Keystone was never deployed, you run kolla-ansible deploy -t nova. What happens?
    Show the answer

    Answer: The run fails at Keystone registration: nothing enforces a dependency when you use tags. Nothing enforces a dependency when you use tags: --tags filters plays, and -t nova on a fresh cloud fails at Keystone registration if Keystone was never deployed CODE. Kolla-Ansible never declares inter-service dependencies explicitly: in 2023.1 no role has a meta directory. Read the page.

    2. You set enable_barbican: yes. The flag changes nova.conf and also common's fluentd and cron files. Does a nova run restart cron, fluentd and kolla_toolbox on the nova hosts?
    Show the answer

    Answer: No: the nova roles have no meta/, so a nova run does not touch them; they restart only when the common play runs. In 2023.1 common is its own play tagged common and the nova roles have no meta/ (no role dependency), so a nova run does not touch cron, fluentd or kolla_toolbox; an untagged or -t common run restarts them on every host of the common play CODE. Read the page.

    3. You run kolla-ansible deploy with --limit set to one new compute host. Why does work still run on control01?
    Show the answer

    Answer: Because delegate_to: groups[x][0] runs work on hosts outside the limit: the Neutron DB create/grant, the bootstrap_neutron container and cell discovery. --limit does not confine actions: delegate_to: groups[x][0] runs work on hosts outside the limit CODE. With deploy --limit set to a compute host, control01 runs the Neutron DB create/grant and the bootstrap_neutron container, plus cell discovery, so "only the new compute" still executes DB-sync containers on a controller. Read the page.

    4. With enable_neutron_agent_ha: yes, the L3 agent wrapper runs neutron-netns-cleanup --force --agent-type l3 before the agent starts. What is established about the routers on that node when neutron_l3_agent restarts?
    Show the answer

    Answer: Nothing: whether they lose north-south and floating-IP connectivity until rebuilt, and whether HA routers fail over, is not established. The cleanup at start is CODE, but whether with the cleanup the routers on that node lose north-south and floating-IP connectivity until rebuilt, and whether HA routers fail over, is NOT AVAILABLE. What neutron-netns-cleanup itself does is neutron code. Read the page.

    5. Kolla-Ansible 2023.1 also accepts ansible-core 2.13. If you run it with 2.13, do the study's statements about Ansible behaviour (loop-notify, tags, run_once with serial) still hold?
    Show the answer

    Answer: Not established: only ansible-core 2.14.18 was read, so the behaviour of 2.13 is not available. Only one ansible-core version was read: Kolla-Ansible 2023.1 also accepts ansible-core 2.13 (tools/kolla-ansible:10-11), and its behaviour is NOT AVAILABLE. The study rests only on reading source code, with no runtime or lab evidence. Read the page.