Three controllers and two computes, nested inside an existing OpenStack cloud¶
A multinode Kolla-Ansible deployment whose hosts are themselves instances of another OpenStack cloud. It gives a real HA control plane (Galera, RabbitMQ cluster, keepalived VIP) and real computes running KVM, without dedicated hardware. It was built and rebuilt many times to reproduce failures on purpose, so it is close to Kolla-Ansible's defaults on everything except what the hosting cloud forces.
Context¶
- Purpose. A throw-away lab, rebuilt often, on which failures are injected and observed. It must behave like a small production control plane (three controllers, clustered services, a VIP) and run real instances.
- Constraints of the hosting cloud, measured before designing anything: project networks with an MTU of 1442; nested virtualisation available inside instances; allowed address pairs and per-port security available; VRRP and UDP 4789 passing between instances; no trunk ports; no Internet egress for ports without a floating IP; outbound TCP 80 blocked; a public address pool that runs out.
- Budget. Roughly 26 vCPU and 64 GB of RAM per lab. One lab at a time.
- Skills. One operator; the deployment had to stay readable as plain Kolla-Ansible.
Topology¶
flowchart LR
deploy["deploy host<br>kolla-ansible venv, registry,<br>HTTPS proxy, NTP"]
subgraph ctl["controllers (x3) also network nodes"]
c1[controller-1]
c2[controller-2]
c3[controller-3]
end
subgraph cmp["computes (x2)"]
k1[compute-1]
k2[compute-2]
end
deploy -- management --> ctl
deploy -- management --> cmp
ctl <-- "api + tunnels (VIP here)" --> cmp
ctl -- external --> ext[(provider network<br>of the lab's Neutron)]
| Role | Count | Size | Runs |
|---|---|---|---|
| Controller | 3 | about 8 vCPU, 16 GB RAM | API services, Galera, RabbitMQ, HAProxy and keepalived, Neutron L3/DHCP/metadata and OVS agents (Kolla-Ansible's default groups put the network services on the controllers) |
| Compute | 2 | about 4 vCPU, 8 GB RAM | nova-compute, libvirt with kvm (nested), OVS agent |
| Deploy host | 1 | existing machine | kolla-ansible in a virtual environment, a local image registry, an HTTPS proxy, an NTP server |
Network design¶
Without trunk ports, every host gets one port per network:
| Network | Hosts | Kolla-Ansible role | Port security |
|---|---|---|---|
| Management (the project's existing network) | all, and the deploy host | SSH and Ansible; egress through the deploy host | enabled, restricted to the deploy host |
| API | all | network_interface (internal API, service traffic) and, by default, tunnel_interface (VXLAN). Carries the internal VIP |
enabled; the VIP allowed on the controller ports with allowed address pairs; IP protocol 112 (VRRP) allowed between members |
| External | controllers only | neutron_external_interface, plugged into the OVS provider bridge |
disabled, because Neutron routers and floating IPs use addresses and MAC addresses the hosting cloud does not know |
-
MTU. The lab networks inherit the hosting cloud's 1442 bytes. Neutron is told so with a configuration override,
/etc/kolla/config/neutron.conf:[DEFAULT] global_physnet_mtu = 1442VXLAN tenant networks then get 1392 bytes. Without it Neutron assumes 1500 and large packets of the instances are silently dropped (see the VM runbook on tunnel MTU mismatch).
-
External interface on the controllers. It must be up but carry no address, persistently, because OVS owns it. A netplan file matching the interface by MAC address with
dhcp4: false,dhcp6: false,accept-ra: false,link-local: [], the lab MTU, andoptional: trueso that boot does not wait for it. - VIP and VRRP. keepalived runs in unicast mode (
keepalived_traffic_mode: "unicast"; the default in theloadbalancerrole is"multicast"), so VRRP needs no multicast in the hosting cloud; Kolla-Ansible writes aunicast_peerlist of the other load balancer hosts' API addresses.keepalived_virtual_router_idis also set to a value unique to the lab: the default51would collide with another lab or any other keepalived on the same network. The lab checks of 2026-10-27 recorded on the VIP timeout page and in the VM runbooks all ran in unicast mode. The first version of this page (2026-10-21) said the default multicast mode passed in this cloud; that was not checked again, and multicast mode has not been reproduced on the lab since.
Storage design¶
- No Cinder, no Ceph: the lab enables Kolla-Ansible's default core services only, and adds optional ones when a reproduction needs them.
- The instance store of each compute is on its own small volume (10 GiB), formatted and mounted by host
preparation on the data directory of Kolla-Ansible's
nova_computeDocker volume (/var/lib/docker/volumes/nova_compute/_data) before Kolla-Ansible deploys. Kolla-Ansible's variables keep their defaults. Without this, a full instance store fills the root file system and takes Docker, the logs and journald down with the instances.
Inventory and configuration layout¶
/etc/kolla/
globals.yml the few values that differ from defaults (below), proxy and registry settings
passwords.yml generated by kolla-genpwd, never committed
config/
neutron.conf global_physnet_mtu
inventory/multinode Kolla-Ansible's multinode inventory: control, network, compute, monitoring, storage, deployment
Values that differ from Kolla-Ansible defaults, and nothing else:
| Setting | Default | Here | Why |
|---|---|---|---|
keepalived_traffic_mode |
multicast |
unicast |
VRRP without depending on multicast in the hosting cloud |
keepalived_virtual_router_id |
51 |
unique per lab | avoid VRRP collisions on a shared network |
global_physnet_mtu (Neutron, override) |
1500 | 1442 | the hosting cloud's MTU |
prechecks_enable_host_os_checks |
true |
false for the deployment group only |
the deploy host runs Ubuntu 22.04, which the 2026.1 prechecks refuse; the lab hosts are still checked |
docker_registry, docker_registry_insecure |
quay.io | the local registry, plain HTTP | pinned images (below) |
docker_http_proxy, docker_https_proxy, container_http_proxy, container_https_proxy and their no_proxy |
unset | the deploy host's proxy | the only egress (below) |
Everything else: Kolla-Ansible defaults (neutron_plugin_agent: openvswitch, nova_compute_virt_type: kvm,
kolla_container_engine: docker).
Key choices and their consequences¶
| Choice | Alternative(s) | Why this one here | Consequence in operation |
|---|---|---|---|
| Hosts are instances of an existing cloud | Bare metal; all-in-one | Cheap, fast to rebuild, real HA | Depends on the hosting cloud's features (nested virt, allowed address pairs, VRRP); if one is withdrawn the design must change |
| One port per network | Trunk ports with VLANs | Trunks not available | More ports per host; interfaces found by MAC address, not by name |
| VIP through allowed address pairs on a plain network | An overlay built between the hosts | Plain Neutron features, verified first | VIP failover depends on VRRP passing in the hosting cloud |
| Local registry with a tag that never moves | Pull from quay.io | quay.io republishes the Kolla tags every night, so two builds would not run the same images | One more service on the deploy host to keep up |
| HTTPS proxy on the deploy host for all egress | A floating IP per host; the router's SNAT | SNAT gave no egress here; port 80 blocked; the floating IP pool ran out | apt, Docker and containers must all get the proxy settings; no_proxy must list the API range, the VIP, the management ranges and the host names, or internal traffic goes through the proxy |
| NTP served by the deploy host | Disable the NTP prechecks | Galera and RabbitMQ need synchronised clocks; NTP does not go through an HTTPS proxy | One more service; the deploy host's clock becomes the lab's reference |
| Network services on the controllers | Dedicated network nodes | Kolla-Ansible's default groups; smaller lab | A controller failure also moves L3 and DHCP agents |
kvm on the computes |
qemu emulation |
Nested virtualisation available; fast instances | If the hosting cloud stops offering nested virt, instances become slow enough to break timeouts |
What went wrong during deployment¶
- Unattended upgrades during
kolla-ansible deploy. The stock Ubuntu 24.04 image enables daily automatic upgrades. Its morning run upgraded 178 packages,libc6andsystemdamong them, on two hosts in the middle of a deploy: systemd restarted services, one container never started and MariaDB never formed its cluster. Fixed by stopping and disablingapt-daily.timerandapt-daily-upgrade.timerand waiting for any running upgrade to end before deploying, on every later build. Details in the deployment error page. - No egress. The project router had SNAT enabled but gave no Internet access to ports without a floating IP;
outbound port 80 was blocked; the public address pool was exhausted.
bootstrap-serversandpullfailed until the HTTPS proxy and the HTTPS-only package sources were in place. - apt failing silently.
apt-get updateonly warns about a source it cannot fetch; running it with--error-on=anyturned a confusing later failure into an immediate, explicit one. - Deploy host refused by the prechecks. 2026.1 accepts only Ubuntu noble among Ubuntu releases, and the prechecks
also run on the deploy host; turning off the host OS check for the
deploymentgroup only kept the check on the lab hosts.
Version notes¶
- 2026.1 (Kolla-Ansible 22.x) requires Python 3.11 or later on the deploy host; the virtual environment uses Python 3.12 installed alongside the system Python, which also covers 2026.2.
- On Ubuntu 24.04,
bootstrap-serversinstalls the Docker SDK from thepython3-dockerpackage (the system Python is externally managed), so pip and its proxy are not involved.
Lessons learned¶
- Measure the hosting cloud before designing the lab: MTU, nested virtualisation, allowed address pairs, VRRP, trunks, egress. Every divergence from Kolla-Ansible defaults in this pattern comes from one of those facts.
- Keep the divergences in one table. The six rows of the table above are the only settings that differ from Kolla-Ansible's defaults here; everything else being default is what makes the lab's behaviour comparable with other deployments.
- Disable automatic upgrades on hosts you deploy to, or schedule them: they will otherwise run in the middle of a deploy one morning.
- Give the instance store its own file system on every compute, in a lab as in production.
Validity and limits¶
Built repeatedly in October 2026 with OpenStack 2026.1, Kolla-Ansible 22.2.0, images 2026.1-ubuntu-noble, Docker,
Open vSwitch, inside one particular hosting cloud. On each build the baseline checks, a VIP failover and a canary
instance per compute passed. Not tried: OVN, Podman, Cinder or Ceph, dedicated network nodes,
other host distributions, other hosting clouds. This is a lab pattern: it is not a recommendation for production,
where the external network, storage and failure domains would be designed very differently.
Related¶
- VM runbook: tunnel MTU mismatch, in
openstack-vm-runbooks(networking/). - Source: lab builds by @soufian-zaouam (OpenStack 2026.1, Kolla-Ansible 22.2.0).
Provenance: written by agent:ozarland-author, from lab builds by @soufian-zaouam on 2026.1. Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your topology, and what happened.