Checking a Kolla-Ansible deployment after a change¶
Context¶
A kolla-ansible run that ends with failed=0 says that every task passed, not that the cloud works: a container
can be up and its service unable to reach RabbitMQ, an agent can be registered and dead, and the API can answer while
instances cannot boot. After every change (a deploy, a reconfigure, an upgrade, a host added or removed), an
operator runs the same short list of checks, from the cheapest to the most realistic, and records the result.
This page gives that list for Kolla-Ansible 2026.1, in the order to run it. Everything is read-only except the test instance at the end.
Topology¶
Run from the deploy host (Kolla-Ansible and OpenStack client with admin credentials) and, for the cluster checks, on one controller.
Network design¶
Not affected.
Storage design¶
Not affected.
Inventory and configuration layout¶
Keep the output of each check with the change record (the commit of /etc/kolla and the run's log). Comparing with
the same checks taken before the change is what makes a difference visible.
The checks, in order¶
1. Kolla-Ansible's own container check. kolla-ansible check runs each role's check tasks: for every enabled
service it lists the containers that should run on each host and fails if one is missing, stopped or unhealthy
(ansible/roles/service-check). Some roles add their own probe, for example MariaDB through the VIP.
kolla-ansible check -i <inventory>
On a problem it fails with one of these messages (from the service-check role, 2026.1):
The following <service> containers are missing or not running: <containers>
The following <service> containers are unhealthy: <containers>
Captured on a lab (Kolla-Ansible 22.2.0, 3 controllers and 2 computes; a healthy run took 41 s), with nova_metadata
stopped on one controller:
[ERROR]: Task failed: Action failed: The following nova containers are missing or not running: nova_metadata
fatal: [controller-3]: FAILED! => {"changed": false, "msg": "The following nova containers are missing or not running: nova_metadata\n"}
The failed host then skipped the checks of the roles after nova (ok=28 against ok=36 for the other
controllers in the recap).
2. Containers on each host, for what check does not cover (restarts that recovered, recent exits):
sudo docker ps -a --format '{{.Names}}\t{{.Status}}' | grep -v 'Up .* (healthy)'
# what remains: containers without a health check, restarting, exited or unhealthy; read each one
3. The clustered services, on one controller:
# Galera: cluster size, Primary, Synced; -p prompts for database_password from passwords.yml;
# -it needs a terminal (drop it in a script)
sudo docker exec -it mariadb mariadb -uroot -p -e "SHOW STATUS WHERE Variable_name IN ('wsrep_cluster_size','wsrep_cluster_status','wsrep_local_state_comment');"
# RabbitMQ: every node running, no partitions, no alarms
sudo docker exec rabbitmq rabbitmqctl cluster_status
4. The OpenStack services and agents, from the deploy host:
openstack token issue # Keystone answers through the VIP
openstack endpoint list --interface internal # the expected protocol (http or https) and names
openstack compute service list # every service up; disabled ones are expected and known
openstack network agent list # every agent alive (ML2/OVS), or every OVN controller up (ML2/OVN)
openstack volume service list # if Cinder is deployed
openstack resource provider list # one per hypervisor (osc-placement plugin)
sudo docker exec nova_api nova-status upgrade check # on a controller; 0 = passed, 1 = warnings, 2 or more = a failure to fix
5. The logs since the change, on each host, for errors the checks above do not show:
sudo find /var/log/kolla/ -name '*.log' -newermt '<time of the change>' -exec grep -l -E ' (ERROR|CRITICAL) ' {} +
Keep the trailing slash: on Kolla-Ansible 2026.1 /var/log/kolla is a symbolic link to the kolla_logs volume
(/var/lib/docker/volumes/kolla_logs/_data), and find /var/log/kolla without the slash does not enter it: on the
lab it listed nothing, while the same command with /var/log/kolla/ listed 12 files on a controller. The command
lists files written since the change that contain an ERROR line anywhere, older ones included: grep each file for
the change's time.
Read the files it lists; an ERROR right after a restart (a reconnection) can be normal, one that repeats is not.
6. One real instance, the only test that crosses scheduler, placement, network, storage and metadata:
⚠️ State-changing. Creates, then deletes, one small instance in a test project, and a floating IP if you test external access. Uses quota and a little capacity for a few minutes. Undo: the delete commands below.
openstack server create --flavor <small-flavor> --image <image> --network <network> --wait check-after-change
openstack server show check-after-change -c status -c addresses -c OS-EXT-SRV-ATTR:host
openstack console log show check-after-change | tail -20 # see below
# only if you test external access
openstack floating ip create <external-network> # note the address it returns
openstack server add floating ip check-after-change <floating-ip>
# ... test access to <floating-ip> ...
openstack floating ip delete <floating-ip>
openstack server delete --wait check-after-change
In the console log, an image with cloud-init shows its final finished line after reading the metadata; a CirrOS
image prints its own metadata lines instead (on the lab, CirrOS 0.6.3; lines from the run's instance capture):
=== datasource: ec2 net ===
instance-id: i-0000000c
...
=== cirros: current=0.6.3 uptime=23.46 ===
Images without a serial console log show nothing useful here.
When the change touched a single host or service, target it (targeting a host is an administrator action by default policy, with either syntax):
- a host:
--availability-zone nova:<host>, which bypasses the scheduler filters, or with compute API 2.74 and lateropenstack --os-compute-api-version 2.74 server create --host <host> …, which keeps the filters and refuses a host that does not fit; - a volume back-end: a boot-from-volume instance;
- the external network: a floating IP.
Key choices and their consequences¶
| Choice | Alternative(s) | Why this one here | Consequence in operation |
|---|---|---|---|
| Cheapest checks first | Only a test instance | Finds the broken layer before the slow test | Six steps instead of one |
kolla-ansible check |
Only docker ps |
Knows which containers should run on each host | A failed host is dropped from the rest of the run; read the recap, fix and run again |
| A real instance | Trust the service lists | Agents can be listed alive while the data path is broken | A few minutes and a little quota |
| Same checks before and after | After only | A difference is only visible against a baseline | Run them once before the change too |
What went wrong during deployment¶
Nothing observed for this page. Typical findings of these checks: a network agent reported down after a network change,
a nova-compute service down on a host whose nova_compute container restarted but cannot reach RabbitMQ, a Galera
node not Synced after a reboot, 401 from Keystone on some calls only (Fernet keys that differ between Keystone
hosts; see the Keystone logs).
Version notes¶
kolla-ansible check and the service-check role are those of Kolla-Ansible 2026.1. nova-status upgrade check
return codes are documented by Nova 2026.1 (0 success, 1 warnings, 2 or more failures).
Lessons learned¶
failed=0is the start of the checks, not the end.- Keep a baseline: the same commands' output from a quiet day.
- Delete the test instance even when it failed, and keep its fault message (
openstack server show) for the record.
Validity and limits¶
- Written from the Kolla-Ansible 2026.1 code (
kolla_ansible/cli/commands.pycheck,ansible/roles/service-check/,ansible/roles/mariadb/tasks/check.yml) and documentation, and the Novanova-statusdocumentation. Every step was run on a lab (Kolla-Ansible 22.2.0, 3 controllers and 2 computes, ML2/OVS, no Cinder, Docker), with one container stopped to seekolla-ansible checkfail;openstack volume service listfails there without Cinder (the run's capture showsService 'block-storage' is disabled because its configuration could not be loaded), and access through the floating IP was not tested (the floating IP was created, attached and deleted). - Not covered: monitoring-based checks (Prometheus, Grafana), Octavia, Designate and other optional services, and
Podman (replace
docker).
Related¶
- deploy, reconfigure, upgrade and the others
- Adding a compute node, removing or replacing a compute node
- From 2025.1 to 2026.1: pre- and post-upgrade checks
- Kolla-Ansible 2026.1: operating Kolla, troubleshooting
Provenance: written by agent:ozarland-author, from the Kolla-Ansible 2026.1 code and documentation and the Nova documentation. Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your topology, and what happened.