Skip to content

🗺️ Production guide

Stop 5 · Go to production

A visual memo for OpenStack operators: one operational lesson per page, to remember, apply and share.

Catalog · How to read a memo · Not covered yet · Build · Contribute

Fence before you evacuate VM stuck in BUILD? Control plane down: what runs?


A field guide, not a manual. Each memo takes one situation an operator meets in production and answers four questions in under a minute: what is happening, what to check first, what not to do yet, and what to remember. The procedure itself, with every command and its limits, stays in the VM runbooks: every memo links to the runbook it was written from.

No imposed tools, no single way of operating OpenStack.

One memo = one operational lesson = one memorable visual.

🧭 How to read a memo

Every memo has the same seven parts, in the same place:

Part What it tells you
Title The situation, as a short question or statement
Situation What you see, and what it means
Diagram The mechanism: where a request stops, which case you are in, what differs between two things that look alike
First checks · green Four things to look at. Always read-only: they never change the platform
Think before acting · amber, red What not to do yet, and the risk of the usual fixes
Operational lesson The one sentence to keep
Go deeper The path of the runbook in the VM runbooks of OzarLand Academy, and how much that runbook was verified

Colour means one thing only, the risk of an action, with the same vocabulary as the OzarLand risk warnings:

On a memo Meaning
🟩 Green Read-only, or the healthy case
🟧 Amber, State-changing. Changes the platform in a bounded way
🟥 Red, Destructive. or Avoid. Can lose data or end workloads, or a shortcut that makes things worse
⬜ Grey, Know this. A fact that changes how you read the situation

Warning

A memo is a memory aid, not a procedure. It is as reliable as the runbook behind it, and it says how reliable that is: the pill at the bottom right shows the runbook's status and release. Today the source runbooks are lab-verified or draft on OpenStack 2026.1, mostly with Kolla-Ansible and ML2/OVS, and none has been confirmed on a production platform by an operator other than its author. A draft runbook's commands were not run by anyone but its author. Each memo's page has a Where this holds paragraph: read it, then read the runbook, before any ⚠️ or 🛑 action.

Catalog

44 memos across 6 categories.

Click a memo for its page: the image, the same content as text, the runbook links and the LinkedIn kit.

VM lifecycle

Memo Operational lesson Runbook
VM stuck in BUILD? PG-01
VM stuck in BUILD?
A networking problem wearing a compute symptom. lifecycle/…
lab-verified
No valid host, with free capacity? PG-02
No valid host, with free capacity?
Three places a build can stop before any compute. Read the exact message first. scheduling/…
lab-verified
Exceeded maximum number of retries? PG-03
Exceeded maximum number of retries?
The fault is a summary. The cause is in the log of each compute that tried. scheduling/…
lab-verified
VM stuck in deleting? PG-04
VM stuck in deleting?
A delete has two paths. Know which one Nova will take before you retry. lifecycle/…
lab-verified
ACTIVE in Nova, frozen for users? PG-05
ACTIVE in Nova, frozen for users?
Nova does not watch the disk. Monitor the instance store and the power state. lifecycle/…
lab-verified
SHUTOFF by itself? PG-21
SHUTOFF by itself?
The scheduler trusts the inventory. Make it match what the host really uses. lifecycle/…
draft
Resize not finished? PG-22
Resize not finished?
VERIFY_RESIZE is not stuck. It waits for a decision only the owner can make. lifecycle/…
lab-verified
Snapshot never finishes? PG-29
Snapshot never finishes?
A slow snapshot is often just a large disk. No error in the compute log? Let it finish. lifecycle/…
lab-verified
Guest broken, Nova says ACTIVE? PG-30
Guest broken, Nova says ACTIVE?
The fault is in the guest, the data is intact. Rescue mode gets you to the disk. lifecycle/…
lab-verified
Console does not open? PG-31
Console does not open?
A console is a chain of hops. The API answer or the proxy log says which one refused. lifecycle/…
draft
Boot from volume fails? PG-32
Boot from volume fails?
The fault names the volume. Read its status and Cinder's message before you retry. scheduling/…
lab-verified
Image too big for the flavor? PG-33
Image too big for the flavor?
The file size is not the disk size. Compare the flavor with the image's virtual size. scheduling/…
lab-verified

Compute operations

Memo Operational lesson Runbook
Compute “down”. Is the host dead? PG-06
Compute “down”. Is the host dead?
“Down” is a missing heartbeat, not a dead host. Prove it before you act. recovery/…
lab-verified
Fence before you evacuate PG-07
Fence before you evacuate
The precondition for evacuation is fencing, not a missing heartbeat. recovery/…
lab-verified
Disabled or down? Two fields PG-08
Disabled or down? Two fields
Disabled is a scheduling decision. Down is a belief about liveness. scheduling/…
lab-verified
Emptying a host for patching? PG-09
Emptying a host for patching?
After a libvirt or QEMU patch, instances can move onto the host, not back. Plan the order. migration/…
lab-verified
Live migration never completes? PG-23
Live migration never completes?
Watch whether the remaining memory goes down. If it hovers, waiting will not help. migration/…
draft
Steal time on pinned CPUs? PG-34
Steal time on pinned CPUs?
Live migration moves the symptom, not the cause. The cause is on the host. migration/…
lab-verified
After a reboot, VMs stay off? PG-35
After a reboot, VMs stay off?
A reboot brings back what is written down. Nova does not restart guests by default. recovery/…
lab-verified
Compute renamed or re-added? PG-36
Compute renamed or re-added?
Three records must agree. A healthy container proves nothing: the log says which one differs. scheduling/…
draft

Networking

Memo Operational lesson Runbook
ACTIVE, but no IP address? PG-10
ACTIVE, but no IP address?
ACTIVE means Nova did its job. It says nothing about the network of the guest. networking/…
lab-verified
Reachable inside, not outside? PG-11
Reachable inside, not outside?
Each check rules out one cause. Go through them in order. networking/…
lab-verified
Ping works, big transfers stall? PG-12
Ping works, big transfers stall?
A green ping proves small packets pass. Test the largest one the network promises. networking/…
lab-verified
One compute's VMs lost network? PG-24
One compute's VMs lost network?
Monitoring built on Nova states cannot see a data-plane outage. networking/…
lab-verified
Security group rule has no effect? PG-25
Security group rule has no effect?
Groups add up, on a default deny. Read the rules before you blame the agent. networking/…
draft
Booted without its SSH key? PG-37
Booted without its SSH key?
An instance gets its identity at first boot. Miss it, and the damage outlives the outage. networking/…
lab-verified
Binding failed for port? PG-38
Binding failed for port?
Nova names the host. Neutron must find a live agent, with the right mappings, under that exact name. networking/…
draft

Storage

Memo Operational lesson Runbook
Volume stuck attaching or detaching? PG-13
Volume stuck attaching or detaching?
The hypervisor tells the truth about the instance. Trust it over both databases. volumes/…
lab-verified
In-use by a VM that is gone? PG-14
In-use by a VM that is gone?
A record is stale only when the hypervisor agrees. Check the host before you change the database. volumes/…
lab-verified
Extended, same size inside? PG-39
Extended, same size inside?
Find the last stage that happened before you repeat the first. volumes/…
draft
Volume stuck in creating? PG-40
Volume stuck in creating?
No host: the scheduler. A host set: the backend. Cinder's message, or its log, says the rest. volumes/…
lab-verified
Volume will not delete? PG-41
Volume will not delete?
A volume that refuses to go may still be in use. Find the user before forcing anything. volumes/…
lab-verified

Control plane

Memo Operational lesson Runbook
Control plane down: what runs? PG-15
Control plane down: what runs?
A control-plane outage is not a workload outage. Protect what runs, then repair. diagnostics/…
lab-verified
MessagingTimeout everywhere? PG-16
MessagingTimeout everywhere?
Computes reported down are a symptom here. Read every RabbitMQ node before you reset anything. recovery/…
lab-verified
APIs fail on database errors? PG-17
APIs fail on database errors?
The data is at risk only during the recovery. Know the most advanced node before you bootstrap. recovery/…
lab-verified
The API answers 503? PG-26
The API answers 503?
A 503 from HAProxy is a messenger. Find out why the backends fail their check. recovery/…
lab-verified
401 with a valid token? PG-27
401 with a valid token?
A fresh token hides the problem. Test with one older than the token cache. recovery/…
lab-verified
Out of database connections? PG-28
Out of database connections?
Three limits, three messages. The message tells you where to look first. recovery/…
draft
Conductor or scheduler down? PG-42
Conductor or scheduler down?
With the conductor gone, every compute looks down. Nothing is wrong on the hosts. recovery/…
lab-verified
Provisioning fails everywhere? PG-43
Provisioning fails everywhere?
When everything fails at once, look at what everything shares. diagnostics/…
draft

Production operations

Memo Operational lesson Runbook
Restarting an agent on a compute? PG-18
Restarting an agent on a compute?
Before a restart, know which process carries the workload, and what keeps it alive. lifecycle/…
lab-verified
A build failed. Which service? PG-19
A build failed. Which service?
The last service that logged the request id is where it stopped. Start there. diagnostics/…
lab-verified
Cleaning up stale records? PG-20
Cleaning up stale records?
Project tools first, the database last. And first of all: is this record true? lifecycle/…
lab-verified
Logs do not line up? PG-44
Logs do not line up?
Before trusting a timeline built from several hosts, compare their clocks. diagnostics/…
lab-verified

📌 Not covered yet

A memo is only written from a runbook that documents the subject. These topics were considered for the first collection and left out, rather than filled with guesses.

No source yet. They need a runbook first, from a real occurrence:

Topic Where it stands
Live migration fails at the pre-check (CPU model, shared storage assumptions) Listed under Pages wanted next in the VM runbooks
Floating IP does not answer with ML2/OVN Same list. The ML2/OVS case is in PG-11
Shelve and unshelve problems Same list
Multi-attach volumes Same list
Storage backend outage: Cinder backend unavailable, storage capacity exhausted Same list ("storage outage"). Storage-backend-only problems are out of the runbooks' scope
Multipath problems Only a log-reading guide exists (draft): not enough for a memo
CPU overcommitment, hypervisor out of resources No runbook. Allocation ratios appear only as a cause in PG-02
Before applying a configuration change, before patching: the general checklist No runbook. PG-09 and PG-18 cover two specific cases

A source exists, a memo does not. Good candidates for a first contribution. This list is generated: it is every runbook, outside the logs/ reading guides, that no memo is written from or cites.

None at the moment: every runbook has its memo.

Memos written from a draft runbook are welcome: the status is printed on the memo.

🗂️ How content is organised

Content and presentation are separate. A memo is a small YAML file; the look comes from one template.

categories/<category>/<slug>.yml   the source of a memo: its text, its diagram, its runbook, its LinkedIn kit
categories/<category>/<slug>.md    generated: the memo's page
templates/                         the template (memo.cjs), its styles (memo.css), the design tokens (tokens.css),
                                   and memo.yml, the starting point of a new memo
assets/                            vendored fonts and icons, the OzarLand mark
scripts/                           build.cjs (validate, render, generate), new-memo.cjs
dist/                              the exported images, 1080 × 1350, ready to post
LINKEDIN.md                        generated: every post title, caption and hashtag set in one place
DESIGN.md                          the visual identity and the rules of the template

🔧 Build it yourself

Requires Node.js 20 or later.

In the directory of this module, docs/production/ of the repository of OzarLand Academy:

npm ci
npx playwright install chromium
npm run build

The build validates every memo, checks each runbook reference against the VM runbooks, docs/runbooks/ next to this module (the path, and that the status and release printed on the memo are the runbook's own), renders the images and regenerates the pages, this catalog and LINKEDIN.md. It fails if a text is over its length budget or if anything does not fit the canvas: the template never shrinks text to make it fit.

Command What it does
npm run new -- <category> <slug> Creates a new memo source from the template, with the next number
npm run build -- --only <slug> Renders one memo while you write it
npm run validate Checks content and runbook references, renders nothing
npm run build -- --scale 2 Exports at 2160 × 2700 (not committed)
npm run build -- --runbooks <dir> Checks the runbook references against another directory than ../runbooks (the VM runbooks of this repository)

🤝 Contributing

How
🐛 Something is wrong, or your platform behaves differently: open an issue. If the runbook is wrong too, the correction belongs there first
💡 Propose a memo: open an issue with the situation, the lesson and the runbook it would rest on
✍️ Write a memo: npm run new, fill in the YAML, build, open a pull request. No design tool needed
🧱 No runbook for your lesson? Share the problem you solved with the Problem solved form first, for a runbook. The memo follows
💬 A question: start a discussion in ozarland, where all OzarLand discussions live

Important

Rules that are not optional: every statement is backed by a runbook; the first checks are read-only; state-changing and destructive actions are flagged; the memo says where it was observed; no secrets, hostnames or addresses. Full guide: CONTRIBUTING.md. Design rules: DESIGN.md.

🌍 Part of OzarLand Academy

This module is the last stop of OzarLand Academy, the learning path of OzarLand: a shared land for better OpenStack operations. Each memo comes from a page of the VM runbooks; the Quick lab is where to try them, and the deployment patterns and Kolla-Ansible Academy explain the platform underneath.

⚖️ License

All content, documentation and code alike, is licensed under the Apache License 2.0 (LICENSE). Reuse it freely with attribution. See licensing. The vendored fonts and icons keep their own licences: see assets/index.md.

OpenStack is a project of the OpenInfra Foundation community. OzarLand is an independent community initiative.