OpenStack on several VMs of an OpenStack cloud (multinode)¶
A multinode OpenStack (2026.1, deployed with Kolla-Ansible) on 3 or 5 VMs of an OpenStack cloud (tested on one cloud so far), in a little over an hour. One command deletes it all. It shows what one VM cannot: instances moving between computes, a compute that dies, and, with 5 VMs, a controller that dies.
Quick start¶
git clone https://github.com/OzarLand/ozarland-academy.git && cd ozarland-academy/docs/lab/variants/openstack-cloud-multinode
cp config.example.env config.env && $EDITOR config.env # OS_CLOUD, NETWORK, SSH_FROM, the two prices; FLOATING_NETWORK if you use one; CONTROLLERS="3" for 5 VMs
./create.sh # shows the cost of all the VMs, asks, creates them (3 to 4 min)
./deploy.sh # deploys OpenStack across them (60 to 70 min)
ssh -i .state/quicklab-multi-1/id_ed25519 ubuntu@<controller-1-address> # then, on controller 1:
source ~/kolla-venv/bin/activate; . /etc/kolla/admin-openrc.sh; openstack server list
./destroy.sh # back on your computer: deletes everything, after asking
SSH_FROMand a dropped connection: as in the single-VM quick start.- Time: plan about 75 minutes (3 VMs) to 90 minutes (5 VMs) before your first exercise: 63 and 73 minutes
measured for
create.shanddeploy.sh, plus filling inconfig.envand logging in.
Read the cost risk first. It runs 3 to 5 VMs, so it costs 2.5 to 4.5 times the single-VM variant. The rest of this page is the detail.
Status: both sizes were run end to end on 2026-10-25, from
create.shtodestroy.sh(figures below);destroy.shleft nothing behind either time.
What you get¶
flowchart TB
you["Your terminal"] -- "SSH (floating IP<br/>of your cloud, optional)" --> ctl1
subgraph cloud["Your OpenStack cloud"]
subgraph net["NETWORK (your project)"]
vip(["VIP<br/>5 VMs only"])
subgraph ctls["Controllers"]
ctl1["Controller 1<br/>deploy host"]
ctl23["Controllers 2 and 3<br/>5 VMs only"]
end
cmps["Compute 1, Compute 2<br/>test instances"]
end
ext["Lab network, LAB_EXT_CIDR<br/>= the lab's public1"]
end
vip -. "on one controller" .- ctls
ctls -- "APIs, instance traffic" --- cmps
ctls -- "lab floating IPs" --- ext
- All the VMs have their address in your project network (
NETWORK). You reach controller 1 on that address, or on a floating IP of your cloud if you setFLOATING_NETWORK. - Controllers run the control plane and the network services. Controller 1 is also the deploy host: you work there.
- Computes run
nova-computeand the test instances. - The lab network (
quicklab-multi-1-ext, rangeLAB_EXT_CIDR) is created by the scripts. It carries the lab's own floating IPs (public1, inside the lab) and is attached to the controllers only. It is not the floating IP of your cloud. - The VIP exists with 3 controllers only: an address of
NETWORKthat keepalived moves between them. With 1 controller, the API address is controller 1's.
The two sizes¶
One setting in config.env, CONTROLLERS, chooses the size: "1" (the default) or "3".
| Size | CONTROLLERS |
VMs | Roles | Required quota |
|---|---|---|---|---|
| Small (default) | "1" |
3 | 1 controller (also the network node and the deploy host), 2 computes | 12 vCPU, 32 GB RAM, 3 instances |
| HA | "3" |
5 | 3 controllers (also network nodes; controller 1 is the deploy host), 2 computes | 20 vCPU, 64 GB RAM, 5 instances |
create.shpicks, for each role, the smallest flavor with at least 4 vCPU and 50 GB of disk: 16 GB of RAM for a controller, 8 GB for a compute. SetCONTROLLER_FLAVORorCOMPUTE_FLAVORto force one.- The quota must also allow: one network and its subnet (the lab network); one port per VM, plus one per controller, plus one more for controller 1 (and one more for the VIP with 3 controllers); one floating IP if you reach the lab from outside the cloud.
The three commands¶
| Script | What it does | Measured, 3 VMs | Measured, 5 VMs |
|---|---|---|---|
create.sh |
Creates the VMs, their key, security group and ports, the lab network (and the VIP port with 3 controllers), and, if you want one, a floating IP for controller 1. Shows the estimated cost per hour and per day for all the VMs and asks before creating anything | 3 min 15 s | 3 min 38 s |
deploy.sh |
Prepares every VM, installs Kolla-Ansible on controller 1 and deploys OpenStack across the VMs: the control plane and the network services on the controller(s), nova-compute on the computes. Ends with a test instance on each compute and a live migration |
59 min 34 s | 69 min 13 s |
destroy.sh |
Deletes everything create.sh created, after asking, and checks that nothing is left |
1 min 14 s | 1 min 53 s |
The cost risk¶
Every VM costs money every hour it exists, used or not, until you run ./destroy.sh. On the reference cloud,
per hour, 3 VMs cost about two and a half times the single-VM variant, and 5 VMs about four and a half times. A lab
forgotten overnight costs that much more too.
| Your cloud | What to do |
|---|---|
| Prepaid | Spending cannot go beyond the credit you loaded. Check your provider's own terms before you rely on it. |
| Pay-as-you-go | Set a budget alert in your provider's console before you start. Destroy the lab the same day. |
create.sh computes the estimate and waits for your confirmation:
COST_PER_HOUR_CONTROLLERandCOST_PER_HOUR_COMPUTE: the price of one VM of each flavor, from your provider's price list. Each is multiplied by the number of VMs of that role.COST_PER_HOUR_EXTRA: what your provider bills apart (root volumes, the floating IP).- It shows the total per hour and per day. It refuses to start while the two prices are empty, unless you run
./create.sh --no-cost-estimate.
What a lab cost on the reference cloud¶
Measured on the reference cloud, a prepaid OpenStack cloud (see the home page). It bills per second. Prices are its pay-as-you-go prices per hour, as listed on 2026-10-23 and checked again on 2026-10-25. The EUR prices are the page's own, not a conversion of the TND ones. The page says nothing about taxes.
| Resource | Used by the lab | TND per hour | EUR per hour |
|---|---|---|---|
Flavor G0.basic.4c16g (4 vCPU, 16 GB, 50 GB disk) |
Each controller | 0.196 | 0.057 |
Flavor G0.basic.4c8g (4 vCPU, 8 GB, 50 GB disk) |
Each compute | 0.140 | 0.041 |
| Floating IP | Only if you set FLOATING_NETWORK |
0.015 | 0.005 |
| 3 VMs (small) | 5 VMs (HA) | |
|---|---|---|
| Per hour | 0.476 TND (0.139 EUR) | 0.868 TND (0.253 EUR) |
| Each day the lab is left running | 11.42 TND (3.34 EUR) | 20.83 TND (6.07 EUR) |
| One run, measured on 2026-10-25 | create.sh to destroy.sh, 65 min 27 s: 0.52 TND (0.15 EUR) |
create.sh to the end of deploy.sh, 72 min 50 s: 1.05 TND (0.31 EUR); a build followed at once by destroy.sh, about 75 min: 1.08 TND (0.32 EUR) |
- The figures for a run are the measured duration multiplied by the listed prices, not an amount read from an invoice.
- Add 0.015 TND (0.005 EUR) per hour if you use a floating IP.
- The exercises add their own time, at the hourly rate above.
- On the reference cloud, set
COST_PER_HOUR_CONTROLLER="0.196",COST_PER_HOUR_COMPUTE="0.140",COST_PER_HOUR_EXTRA="0"("0.015"with a floating IP) andCURRENCY="TND", or the EUR prices withCURRENCY="EUR".
Safety rules¶
The same as the single-VM variant:
- A cost estimate is required.
create.shrefuses to start withoutCOST_PER_HOUR_CONTROLLERandCOST_PER_HOUR_COMPUTE, unless you run./create.sh --no-cost-estimate. - Nothing is created before you confirm.
create.shlists what it will create, with the estimated cost per hour and per day, and waits for your answer. - Every resource is named from the prefix
${PREFIX}${LAB}-(defaultquicklab-multi-1-). See how resources are named. - 🛑
destroy.shdeletes everything in the project whose name starts with${PREFIX}${LAB}-, after listing it and asking. KeepPREFIXspecific to the lab: never a prefix that other resources of your project share. The scripts refuse aPREFIXthat does not start withquicklabunless you setALLOW_ANY_PREFIX=yesinconfig.env. - Destroy checks that nothing is left. It lists every kind of resource again after deleting, and fails loudly if anything of the lab is still there.
What you need¶
- An account on an OpenStack cloud, with the quota of the size you choose (above).
- The same tools,
clouds.yaml, network and access as the single-VM variant: see its what you need. - The right to create a network, a subnet and ports with port security disabled in your project: the lab's floating IPs live on a network the scripts create.
- Nested virtualisation on the hosting cloud, so that instances on the lab's computes run with KVM. Live migration
between two computes works with
qemutoo, but is much slower. - For the HA size, check before you pay for 5 VMs:
- The hosting cloud must let the controllers share a virtual IP (allowed address pairs on their ports).
openstack extension list --network | grep -i allowed-address-pairson your account tells you, if your provider's documentation does not. - It must pass VRRP (IP protocol 112) between them. Only your provider or a test between two VMs can tell. The scripts use keepalived in unicast mode, so multicast is not needed.
- The pattern's context lists what was measured on the hosting cloud before designing the lab. Its network design shows how the virtual IP and VRRP are set up.
- The hosting cloud must let the controllers share a virtual IP (allowed address pairs on their ports).
Steps¶
cp config.example.env config.env
$EDITOR config.env # OS_CLOUD, NETWORK, SSH_FROM, the two prices; CONTROLLERS="3" for the HA size
./create.sh # shows what it will create and the cost for all the VMs, asks, creates
./deploy.sh # about an hour; can be run again after a failure, it resumes
ssh -i .state/<prefix><lab>/id_ed25519 ubuntu@<controller-1-address>
source ~/kolla-venv/bin/activate; . /etc/kolla/admin-openrc.sh
openstack server list # demo1 and demo2, one on each compute
./destroy.sh # when you are done: deletes everything, after asking
<controller-1-address> is controller 1's floating IP if you set FLOATING_NETWORK, its address in NETWORK
otherwise. deploy.sh prints the full ssh command at the end.
How resources are named¶
Everything starts with ${PREFIX}${LAB}-, quicklab-multi-1- by default:
| Resource | Name |
|---|---|
| VMs | quicklab-multi-1-ctl-1 (and -ctl-2, -ctl-3 with 3 controllers), quicklab-multi-1-cmp-1 and -cmp-2 |
| Ports | <vm>-port on every VM; <vm>-ext on the controllers; quicklab-multi-1-ctl-1-gw; quicklab-multi-1-vip with 3 controllers |
| Lab network | quicklab-multi-1-ext and its subnet |
| Security group, key | quicklab-multi-1-sg, quicklab-multi-1-key |
| Root volumes (only when a flavor has no disk) | <vm>-root |
| Floating IP | Carries the lab's name in its description |
destroy.sh finds them all by these names, even if the creation failed half-way or .state/ was lost.
Inside the lab¶
Work on controller 1: it is the deploy host, and the OpenStack client and the admin credentials are there.
- The computes' names in Nova are their VM names (
quicklab-multi-1-cmp-1,-cmp-2by default):openstack compute service list --service nova-computelists them. - To log in to another VM of the lab from controller 1, use the lab's key, which
deploy.shcopied there:ssh -i ~/.ssh/quicklab_ed25519 ubuntu@<address-in-NETWORK>. The addresses are in.state/<prefix><lab>/state.envon your computer (ADDR_cmp_1,ADDR_cmp_2, …), and in the Kolla-Ansible inventory~/multinodeon controller 1. - The test resources are those of the single-VM variant (
cirros,m1.tiny,demo-net,public1,mykey; see its inside the lab), with two differences:demo-netis10.98.0.0/24;- there are two test instances,
demo1anddemo2, one on each compute. The smoke test then live-migratesdemo1, so both may end on the same compute.
- The floating IPs of
public1are reachable from controller 1. - Horizon:
deploy.shprints the SSH tunnel command at the end, as in the single-VM variant.
What it lets you practise¶
| With 3 VMs (two computes) | With 5 VMs (three controllers), in addition |
|---|---|
| Live migration of a running instance from one compute to the other, without stopping it | Losing a controller, and what the other two keep doing |
| A compute that goes down: fence it (make sure it is really off) before you evacuate its instances to the other compute | The Galera (MariaDB) and RabbitMQ clusters: their state, a member that leaves and rejoins |
| Host maintenance: disable a compute, drain it by migrating its instances away, work on it, put it back | VIP failover: keepalived moves the API's virtual IP to another controller |
| A build rescheduled on another compute: when the first compute chosen fails the build, Nova retries on the other | Quorum recovery: bringing a cluster back when too many members were lost at once |
The exercises written for this variant are listed in exercises/, marked
multinode variant.
How the lab is built¶
The design follows the pattern Three controllers and two computes, nested inside an existing OpenStack cloud. Read it for the reasoning and the errors met while building such a lab; this page does not repeat it. Where the scripts differ from that pattern:
| Pattern | These scripts | Why |
|---|---|---|
| A separate deploy host, with its own registry, proxy and NTP | Controller 1 is the deploy host; images from quay.io (or your DOCKER_REGISTRY), your HTTPS_PROXY and NTP_SERVER if you need them |
One VM fewer to pay for |
| Controllers of about 8 vCPU and 16 GB | Controllers of 4 vCPU and 16 GB (computes as in the pattern: 4 vCPU, 8 GB) | The smallest flavors that ran both sizes; set CONTROLLER_FLAVOR for bigger controllers |
| keepalived in multicast | keepalived in unicast (3 controllers) | Some cloud networks drop multicast |
| A dedicated volume for each compute's instance store | The compute's own disk, at Kolla-Ansible's default location | Simpler; a full instance store can then fill the compute's root file system |
| 1 controller: not covered | No HAProxy, no keepalived, no ProxySQL; the API address is controller 1's | One controller needs no load balancer |
Releases¶
Last run on 2026-10-25, both sizes, on the reference cloud: OpenStack 2026.1, Kolla-Ansible 22.2.0, ansible-core 2.20.10, images from quay.io through an HTTPS proxy, Ubuntu 24.04 VMs with nested KVM.
Step of deploy.sh |
3 VMs | 5 VMs |
|---|---|---|
| Prepare every VM (in parallel) | 3 min 20 s | 5 min 28 s |
| Install Kolla-Ansible on controller 1 | 1 min 05 s | 1 min 16 s |
| Write the configuration | 7 s | 11 s |
bootstrap-servers |
4 min 22 s | 4 min 40 s |
prechecks |
48 s | 1 min 45 s |
pull (images from quay.io) |
18 min 48 s | 21 min 03 s |
deploy |
22 min 06 s | 30 min 37 s |
post-deploy, client (and osc-placement since the 5-VM run) |
24 s | 26 s |
| Smoke test (an instance on each compute, ping and SSH, a live migration) | 8 min 32 s | 3 min 50 s |
deploy.sh in all |
59 min 34 s | 69 min 13 s |
- The 5-VM size was run twice; the other run took 70 minutes from
create.shto the end ofdeploy.sh. - On the 5-VM lab, a manual check after the smoke test (not part of
deploy.sh) found the VIP held by one controller, Galera with 3 nodesSyncedandPrimary, RabbitMQ with 3 running nodes, HAProxy's Keystone backendsUPon the three controllers, and Nova's and Neutron's agents alive on all three. - The live migration of the smoke test lost, to a ping every 0.2 s: 8.6 s on the 3-VM run, 2.8 s and 2.4 s on the two 5-VM runs.
When something fails¶
deploy.shprints the failing task ofkolla-ansible. The logs stay on controller 1, in~/.quicklab/.- Fix the cause and run
./deploy.shagain: it skips the steps already done. - To start again from scratch:
./destroy.sh, then./create.sh.
What it does not include¶
- High availability with 3 VMs: one controller is a single point of failure. Losing it stops the APIs and, since the network services run on it too, routing and DHCP for the instances. Use the 5-VM size to practise HA.
- Shared storage: no Cinder, no Ceph. Instances live on the local disk of their compute. Live migration copies the disk (block migration), and evacuation rebuilds the instance from its image: the data on its disk is lost.
- Load balancing (Octavia), production use, or support. It is a learning lab, deleted after use.
- More than two computes by default (
COMPUTESaccepts up to 4, not tested): enough to move instances, not to practise capacity planning.
Done for the day? Run ./destroy.sh: every VM costs money every hour it exists. Then check that nothing is left:
delete everything when you are done.