🗺️ Production guide¶
A visual memo for OpenStack operators: one operational lesson per page, to remember, apply and share.
Catalog · How to read a memo · Not covered yet · Build · Contribute
A field guide, not a manual. Each memo takes one situation an operator meets in production and answers four questions in under a minute: what is happening, what to check first, what not to do yet, and what to remember. The procedure itself, with every command and its limits, stays in the VM runbooks: every memo links to the runbook it was written from.
No imposed tools, no single way of operating OpenStack.
One memo = one operational lesson = one memorable visual.
🧭 How to read a memo¶
Every memo has the same seven parts, in the same place:
| Part | What it tells you |
|---|---|
| Title | The situation, as a short question or statement |
| Situation | What you see, and what it means |
| Diagram | The mechanism: where a request stops, which case you are in, what differs between two things that look alike |
| First checks · green | Four things to look at. Always read-only: they never change the platform |
| Think before acting · amber, red | What not to do yet, and the risk of the usual fixes |
| Operational lesson | The one sentence to keep |
| Go deeper | The path of the runbook in the VM runbooks of OzarLand Academy, and how much that runbook was verified |
Colour means one thing only, the risk of an action, with the same vocabulary as the OzarLand risk warnings:
| On a memo | Meaning | |
|---|---|---|
| 🟩 | Green | Read-only, or the healthy case |
| 🟧 | Amber, State-changing. | Changes the platform in a bounded way |
| 🟥 | Red, Destructive. or Avoid. | Can lose data or end workloads, or a shortcut that makes things worse |
| ⬜ | Grey, Know this. | A fact that changes how you read the situation |
Warning
A memo is a memory aid, not a procedure. It is as reliable as the runbook behind it, and it says how reliable
that is: the pill at the bottom right shows the runbook's status and release. Today the source runbooks are
lab-verified or draft on OpenStack 2026.1, mostly with Kolla-Ansible and ML2/OVS, and none has been confirmed
on a production platform by an operator other than its author. A draft runbook's commands were not run by anyone
but its author. Each memo's page has a Where this holds paragraph:
read it, then read the runbook, before any ⚠️ or 🛑 action.
Catalog¶
44 memos across 6 categories.
Click a memo for its page: the image, the same content as text, the runbook links and the LinkedIn kit.
VM lifecycle¶
| Memo | Operational lesson | Runbook | |
|---|---|---|---|
![]() |
PG-01 VM stuck in BUILD? |
A networking problem wearing a compute symptom. | lifecycle/…lab-verified |
![]() |
PG-02 No valid host, with free capacity? |
Three places a build can stop before any compute. Read the exact message first. | scheduling/…lab-verified |
![]() |
PG-03 Exceeded maximum number of retries? |
The fault is a summary. The cause is in the log of each compute that tried. | scheduling/…lab-verified |
![]() |
PG-04 VM stuck in deleting? |
A delete has two paths. Know which one Nova will take before you retry. | lifecycle/…lab-verified |
![]() |
PG-05 ACTIVE in Nova, frozen for users? |
Nova does not watch the disk. Monitor the instance store and the power state. | lifecycle/…lab-verified |
![]() |
PG-21 SHUTOFF by itself? |
The scheduler trusts the inventory. Make it match what the host really uses. | lifecycle/…draft |
![]() |
PG-22 Resize not finished? |
VERIFY_RESIZE is not stuck. It waits for a decision only the owner can make. | lifecycle/…lab-verified |
![]() |
PG-29 Snapshot never finishes? |
A slow snapshot is often just a large disk. No error in the compute log? Let it finish. | lifecycle/…lab-verified |
![]() |
PG-30 Guest broken, Nova says ACTIVE? |
The fault is in the guest, the data is intact. Rescue mode gets you to the disk. | lifecycle/…lab-verified |
![]() |
PG-31 Console does not open? |
A console is a chain of hops. The API answer or the proxy log says which one refused. | lifecycle/…draft |
![]() |
PG-32 Boot from volume fails? |
The fault names the volume. Read its status and Cinder's message before you retry. | scheduling/…lab-verified |
![]() |
PG-33 Image too big for the flavor? |
The file size is not the disk size. Compare the flavor with the image's virtual size. | scheduling/…lab-verified |
Compute operations¶
| Memo | Operational lesson | Runbook | |
|---|---|---|---|
![]() |
PG-06 Compute “down”. Is the host dead? |
“Down” is a missing heartbeat, not a dead host. Prove it before you act. | recovery/…lab-verified |
![]() |
PG-07 Fence before you evacuate |
The precondition for evacuation is fencing, not a missing heartbeat. | recovery/…lab-verified |
![]() |
PG-08 Disabled or down? Two fields |
Disabled is a scheduling decision. Down is a belief about liveness. | scheduling/…lab-verified |
![]() |
PG-09 Emptying a host for patching? |
After a libvirt or QEMU patch, instances can move onto the host, not back. Plan the order. | migration/…lab-verified |
![]() |
PG-23 Live migration never completes? |
Watch whether the remaining memory goes down. If it hovers, waiting will not help. | migration/…draft |
![]() |
PG-34 Steal time on pinned CPUs? |
Live migration moves the symptom, not the cause. The cause is on the host. | migration/…lab-verified |
![]() |
PG-35 After a reboot, VMs stay off? |
A reboot brings back what is written down. Nova does not restart guests by default. | recovery/…lab-verified |
![]() |
PG-36 Compute renamed or re-added? |
Three records must agree. A healthy container proves nothing: the log says which one differs. | scheduling/…draft |
Networking¶
| Memo | Operational lesson | Runbook | |
|---|---|---|---|
![]() |
PG-10 ACTIVE, but no IP address? |
ACTIVE means Nova did its job. It says nothing about the network of the guest. | networking/…lab-verified |
![]() |
PG-11 Reachable inside, not outside? |
Each check rules out one cause. Go through them in order. | networking/…lab-verified |
![]() |
PG-12 Ping works, big transfers stall? |
A green ping proves small packets pass. Test the largest one the network promises. | networking/…lab-verified |
![]() |
PG-24 One compute's VMs lost network? |
Monitoring built on Nova states cannot see a data-plane outage. | networking/…lab-verified |
![]() |
PG-25 Security group rule has no effect? |
Groups add up, on a default deny. Read the rules before you blame the agent. | networking/…draft |
![]() |
PG-37 Booted without its SSH key? |
An instance gets its identity at first boot. Miss it, and the damage outlives the outage. | networking/…lab-verified |
![]() |
PG-38 Binding failed for port? |
Nova names the host. Neutron must find a live agent, with the right mappings, under that exact name. | networking/…draft |
Storage¶
| Memo | Operational lesson | Runbook | |
|---|---|---|---|
![]() |
PG-13 Volume stuck attaching or detaching? |
The hypervisor tells the truth about the instance. Trust it over both databases. | volumes/…lab-verified |
![]() |
PG-14 In-use by a VM that is gone? |
A record is stale only when the hypervisor agrees. Check the host before you change the database. | volumes/…lab-verified |
![]() |
PG-39 Extended, same size inside? |
Find the last stage that happened before you repeat the first. | volumes/…draft |
![]() |
PG-40 Volume stuck in creating? |
No host: the scheduler. A host set: the backend. Cinder's message, or its log, says the rest. | volumes/…lab-verified |
![]() |
PG-41 Volume will not delete? |
A volume that refuses to go may still be in use. Find the user before forcing anything. | volumes/…lab-verified |
Control plane¶
| Memo | Operational lesson | Runbook | |
|---|---|---|---|
![]() |
PG-15 Control plane down: what runs? |
A control-plane outage is not a workload outage. Protect what runs, then repair. | diagnostics/…lab-verified |
![]() |
PG-16 MessagingTimeout everywhere? |
Computes reported down are a symptom here. Read every RabbitMQ node before you reset anything. | recovery/…lab-verified |
![]() |
PG-17 APIs fail on database errors? |
The data is at risk only during the recovery. Know the most advanced node before you bootstrap. | recovery/…lab-verified |
![]() |
PG-26 The API answers 503? |
A 503 from HAProxy is a messenger. Find out why the backends fail their check. | recovery/…lab-verified |
![]() |
PG-27 401 with a valid token? |
A fresh token hides the problem. Test with one older than the token cache. | recovery/…lab-verified |
![]() |
PG-28 Out of database connections? |
Three limits, three messages. The message tells you where to look first. | recovery/…draft |
![]() |
PG-42 Conductor or scheduler down? |
With the conductor gone, every compute looks down. Nothing is wrong on the hosts. | recovery/…lab-verified |
![]() |
PG-43 Provisioning fails everywhere? |
When everything fails at once, look at what everything shares. | diagnostics/…draft |
Production operations¶
| Memo | Operational lesson | Runbook | |
|---|---|---|---|
![]() |
PG-18 Restarting an agent on a compute? |
Before a restart, know which process carries the workload, and what keeps it alive. | lifecycle/…lab-verified |
![]() |
PG-19 A build failed. Which service? |
The last service that logged the request id is where it stopped. Start there. | diagnostics/…lab-verified |
![]() |
PG-20 Cleaning up stale records? |
Project tools first, the database last. And first of all: is this record true? | lifecycle/…lab-verified |
![]() |
PG-44 Logs do not line up? |
Before trusting a timeline built from several hosts, compare their clocks. | diagnostics/…lab-verified |
📌 Not covered yet¶
A memo is only written from a runbook that documents the subject. These topics were considered for the first collection and left out, rather than filled with guesses.
No source yet. They need a runbook first, from a real occurrence:
| Topic | Where it stands |
|---|---|
| Live migration fails at the pre-check (CPU model, shared storage assumptions) | Listed under Pages wanted next in the VM runbooks |
| Floating IP does not answer with ML2/OVN | Same list. The ML2/OVS case is in PG-11 |
| Shelve and unshelve problems | Same list |
| Multi-attach volumes | Same list |
| Storage backend outage: Cinder backend unavailable, storage capacity exhausted | Same list ("storage outage"). Storage-backend-only problems are out of the runbooks' scope |
| Multipath problems | Only a log-reading guide exists (draft): not enough for a memo |
| CPU overcommitment, hypervisor out of resources | No runbook. Allocation ratios appear only as a cause in PG-02 |
| Before applying a configuration change, before patching: the general checklist | No runbook. PG-09 and PG-18 cover two specific cases |
A source exists, a memo does not. Good candidates for a first contribution. This list is generated: it is every
runbook, outside the logs/ reading guides, that no memo is written from or cites.
None at the moment: every runbook has its memo.
Memos written from a draft runbook are welcome: the status is printed on the memo.
🗂️ How content is organised¶
Content and presentation are separate. A memo is a small YAML file; the look comes from one template.
categories/<category>/<slug>.yml the source of a memo: its text, its diagram, its runbook, its LinkedIn kit
categories/<category>/<slug>.md generated: the memo's page
templates/ the template (memo.cjs), its styles (memo.css), the design tokens (tokens.css),
and memo.yml, the starting point of a new memo
assets/ vendored fonts and icons, the OzarLand mark
scripts/ build.cjs (validate, render, generate), new-memo.cjs
dist/ the exported images, 1080 × 1350, ready to post
LINKEDIN.md generated: every post title, caption and hashtag set in one place
DESIGN.md the visual identity and the rules of the template
🔧 Build it yourself¶
Requires Node.js 20 or later.
In the directory of this module, docs/production/ of the repository of OzarLand Academy:
npm ci
npx playwright install chromium
npm run build
The build validates every memo, checks each runbook reference against the VM runbooks, docs/runbooks/ next to this module
(the path, and that the status and release printed on the memo are the runbook's own), renders the images and
regenerates the pages, this catalog and LINKEDIN.md. It fails if a text is over its length budget or if anything
does not fit the canvas: the template never shrinks text to make it fit.
| Command | What it does |
|---|---|
npm run new -- <category> <slug> |
Creates a new memo source from the template, with the next number |
npm run build -- --only <slug> |
Renders one memo while you write it |
npm run validate |
Checks content and runbook references, renders nothing |
npm run build -- --scale 2 |
Exports at 2160 × 2700 (not committed) |
npm run build -- --runbooks <dir> |
Checks the runbook references against another directory than ../runbooks (the VM runbooks of this repository) |
🤝 Contributing¶
| How | |
|---|---|
| 🐛 | Something is wrong, or your platform behaves differently: open an issue. If the runbook is wrong too, the correction belongs there first |
| 💡 | Propose a memo: open an issue with the situation, the lesson and the runbook it would rest on |
| ✍️ | Write a memo: npm run new, fill in the YAML, build, open a pull request. No design tool needed |
| 🧱 | No runbook for your lesson? Share the problem you solved with the Problem solved form first, for a runbook. The memo follows |
| 💬 | A question: start a discussion in ozarland, where all OzarLand discussions live |
Important
Rules that are not optional: every statement is backed by a runbook; the first checks are read-only; state-changing and destructive actions are flagged; the memo says where it was observed; no secrets, hostnames or addresses. Full guide: CONTRIBUTING.md. Design rules: DESIGN.md.
🌍 Part of OzarLand Academy¶
This module is the last stop of OzarLand Academy, the learning path of OzarLand: a shared land for better OpenStack operations. Each memo comes from a page of the VM runbooks; the Quick lab is where to try them, and the deployment patterns and Kolla-Ansible Academy explain the platform underneath.
⚖️ License¶
All content, documentation and code alike, is licensed under the Apache License 2.0 (LICENSE). Reuse it freely with attribution. See licensing. The vendored fonts and icons keep their own licences: see assets/index.md.
OpenStack is a project of the OpenInfra Foundation community. OzarLand is an independent community initiative.











































