Where the OpenStack logs are, and how to read one log line¶
Every investigation starts with two questions: where does this service write, and what does each field of its line
mean. The answers depend on how the platform was deployed, and much less on the service, because almost every
OpenStack service logs through the same library, oslo.log, with the same line format.
This page covers:
- where the logs are on a Kolla-Ansible deployment and on a package (distribution) install;
- the default
oslo.logline, field by field, including the request ids and the instance prefix; - the log levels, and how to turn debug on for one service and back off;
- the logs that do not use
oslo.log(RabbitMQ, MariaDB, HAProxy, Open vSwitch, OVN, libvirt), with their own formats.
How to follow one request across services is on its own page: what a VM creation traverses, and tracing by request id.
Where the logs are¶
Kolla-Ansible¶
Every container writes into one Docker (or Podman) volume, kolla_logs, mounted at /var/log/kolla in the
containers. On the host, /var/log/kolla points to that volume, one directory per service
(Kolla-Ansible troubleshooting guide):
# On any node: where the log volume really is (sudo: /var/lib/docker is readable by root only)
sudo readlink -f /var/log/kolla
# Kolla-Ansible's guide shows /var/lib/docker/volumes/kolla_logs/_data
sudo ls /var/log/kolla/
The file names come from the log_file and log_dir options that Kolla-Ansible 2026.1 writes into each service's
configuration. With log_dir alone, oslo.log names the file after the program: <log_dir>/<program>.log
(_get_log_file_path).
| Service | Container (default name) | Log file | Where it runs |
|---|---|---|---|
| nova-api | nova_api |
/var/, requests in nova-api-uwsgi.log |
controllers |
| nova-scheduler | nova_scheduler |
/var/ |
controllers |
| nova-conductor | nova_conductor |
/var/ (cell conductor) |
controllers |
| nova super conductor | nova_super_conductor |
/var/ (only with enable_cells: true) |
controllers |
| nova-compute | nova_compute |
/var/ |
computes |
| libvirt | nova_libvirt |
/var/; each instance's QEMU log is libvirt's /var/ inside the container (not under kolla_logs, see the limits) |
computes |
| placement-api | placement_api |
/var/, requests in placement-api-uwsgi.log |
controllers |
| neutron-server (API) | neutron_server |
/var/, requests in neutron-server-uwsgi.log |
controllers |
| Neutron RPC server, periodic workers | neutron_rpc_server, neutron_periodic_worker |
/var/, neutron-periodic-workers.log (named after the program) |
controllers |
| Neutron agents | neutron_openvswitch_agent, neutron_l3_agent, neutron_dhcp_agent, neutron_metadata_agent, neutron_ovn_metadata_agent |
/var/ |
network nodes, computes |
| keystone | keystone |
/var/ |
controllers |
| glance-api | glance_api |
/var/ |
controllers |
| cinder | cinder_api, cinder_scheduler, cinder_volume |
/var/, cinder-scheduler.log, cinder-volume.log |
controllers, storage nodes |
| RabbitMQ | rabbitmq |
/var/ |
controllers |
| MariaDB / Galera | mariadb |
/var/ |
controllers |
| HAProxy | haproxy |
under /var/ (sent through syslog to fluentd, file haproxy_latest) |
load balancer nodes |
| ProxySQL (database proxy, default with MariaDB) | proxysql |
/var/ (errorlog in proxysql.yaml.j2) |
load balancer nodes |
| Open vSwitch | openvswitch_vswitchd, openvswitch_db |
/var/, ovsdb-server.log |
computes, network nodes |
| OVN | ovn_controller, ovn_northd, ovn_nb_db, ovn_sb_db |
/var/, ovn-northd.log, ovn-nb-db.log, ovn-sb-db.log |
computes, controllers |
Sources for the table: the 2026.1 templates nova.conf.j2 (nova and nova-cell roles), neutron.conf.j2,
placement.conf.j2, keystone.conf.j2, glance-api.conf.j2, cinder.conf.j2, the service-uwsgi-config role
(logto2 = /var/log/kolla/<role>/<service>-uwsgi.log), and fluentd's input definitions, which list the files it
reads (input/02-mariadb.conf.j2, 03-rabbitmq.conf.j2, 05-libvirt.conf.j2, 10-openvswitch.conf.j2) and
where it writes HAProxy's syslog stream (syslog_facilities in the fluentd role defaults). Directory and file
names can differ when a service is disabled, renamed or overridden; ls /var/log/kolla/<service>/ is the truth on
your platform.
Things that surprise people on Kolla-Ansible:
docker logsis mostly empty. Most containers do not log to standard output, sodocker logs <container>shows only the start-up of the container (the configuration copy bykolla_set_configs) and crashes beforeoslo.logis set up. Read the files instead;docker logsis useful when a container restarts in a loop.- The API services run under uWSGI. In 2026.1 the API services (Nova, Neutron, Placement, Glance, Cinder,
Keystone and others) write two files: the application log (
<service>.log,oslo.logformat) and the uWSGI log (<service>-uwsgi.log), which carries one line per HTTP request in uWSGI's own format, starting with[pid: <pid>|app: <n>|req: <n>/<n>] <client address> .... - fluentd reads the files, it does not replace them. The
fluentdcontainer (enabled by default,enable_fluentd: true) tails the files above. It reads the systemd journal only with central logging (enable_fluentd_systemdisenable_fluentd and enable_central_loggingingroup_vars/all/fluentd.yml), and it sends logs to OpenSearch only whenenable_central_loggingis true (defaultfalse). Without central logging, the files on each node are all there is. - The host's own logs are outside
/var/log/kolla. The kernel, the container engine, chrony and systemd units log to the journal of the host (journalctl), as on any Linux host.
Kolla-Ansible: read a service's log on the node where it runs:
# sudo: the files are readable by root only, and so is the Docker socket
sudo tail -f /var/log/kolla/nova/nova-compute.log
sudo grep -nE ' [0-9]+ (ERROR|CRITICAL) ' /var/log/kolla/nova/nova-compute.log | tail -20
# the container engine's view of one container: start-up, crashes, restarts
sudo docker ps -a --filter name=nova_compute
sudo docker logs --tail 50 nova_compute # podman logs on a Podman deployment
Package installs (distribution packages)¶
(Not run on the lab: the lab was Kolla-Ansible.)
Nova's documentation gives /var/log/nova/ as the directory of its log files, one per service, with service
names that differ by distribution
(Compute log files):
| Log file | Service on RHEL, CentOS, Fedora | Service on Ubuntu, Debian |
|---|---|---|
nova-api.log |
openstack-nova-api |
nova-api |
nova-compute.log |
openstack-nova-compute |
nova-compute |
nova-conductor.log |
openstack-nova-conductor |
nova-conductor |
nova-scheduler.log |
openstack-nova-scheduler |
nova-scheduler |
The other projects follow the same pattern (/var/log/neutron/, /var/log/cinder/, and so on), but where a
service writes depends on its packaging, not on OpenStack. oslo.log adds one handler for each output that is
configured, and they add up: a service can write to a file and to the journal at the same time
(_setup_logging_from_conf):
log_fileand/orlog_dirset (in the configuration file, or as--log-fileon the command line of the unit): a file;use_stderr = true: standard error;use_journal = true: the systemd journal, natively, with structured fields;use_syslog = true: syslog;- none of the first three: standard output, which systemd captures in the journal.
So on a package install, look in both places:
Any:
# What the unit runs, with which --log-file or --config-file
systemctl cat <unit> # e.g. openstack-nova-compute or nova-compute
# What the journal holds for it
journalctl -u <unit> --since "1 hour ago"
# The configured log options
grep -E '^\s*(log_file|log_dir|use_journal|use_syslog|use_stderr|debug)\s*=' /etc/nova/nova.conf
When a service logs natively to the journal (use_journal = true), each entry carries structured fields set by
oslo.log's journal handler: SYSLOG_IDENTIFIER (the program name), PRIORITY, LOGGER_NAME,
LOGGER_LEVEL, CODE_FILE, CODE_LINE, CODE_FUNC, and, when known, REQUEST_ID, PROJECT_ID,
PROJECT_NAME, USER_ID, USER_NAME, plus EXCEPTION_INFO for a traceback
(OSJournalHandler).
That lets you filter by field instead of by text:
journalctl REQUEST_ID=<request-id> # not run on the lab
journalctl SYSLOG_IDENTIFIER=nova-compute PRIORITY=3 # errors only (3 = err)
These fields exist only with use_journal = true. A service that writes to standard output, captured by
systemd, has plain text lines in the journal and none of these fields.
Reading one oslo.log line¶
The default format¶
oslo.log uses one of two formats, chosen per line
(ContextFormatter.format):
-
when the line was logged inside a request (it has a request id),
logging_context_format_string, whose default in 2026.1 is%(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(user_identity)s] %(instance)s%(message)s -
otherwise (periodic tasks, start-up, background threads),
logging_default_format_string:%(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [-] %(instance)s%(message)s
Both defaults are in
oslo_log/_options.py.
A deployment can change them in the [DEFAULT] section of each service, so check that first if your lines do not
look like this. Kolla-Ansible 2026.1 does not override them: its fluentd parser expects exactly this shape (the
format1 expression in
input/00-global.conf.j2).
Field by field¶
The shape of a line logged inside a request, with placeholders (this is the format above, not a captured line):
<date> <time>.<ms> <pid> <LEVEL> <module> [<global-request-id or None> req-<uuid> <user-id> <project-id> <domain-id> <system-scope> <user-domain-id> <project-domain-id>] [instance: <instance-uuid>] <message>
| Field | Format element | What it tells you |
|---|---|---|
<date> <time>.<ms> |
%(asctime)s.%(msecs)03d, date format %Y-%m-%d %H:%M:%S |
The local time of the host that wrote the line, without a time zone. Comparing hosts assumes their clocks agree (clock skew). |
<pid> |
%(process)d |
The process id. An API or a conductor runs several workers: lines of one request stay in one worker; a restart shows as a new pid. |
<LEVEL> |
%(levelname)s |
DEBUG, INFO, WARNING, ERROR, CRITICAL (see below). |
<module> |
%(name)s |
The Python logger, usually the module that logged: nova.compute.manager, nova.scheduler.manager, oslo_messaging._drivers.impl_rabbit. It tells you which part of the code speaks, and whether it is the service itself or a library (oslo_messaging, oslo_db, keystonemiddleware). |
<global-request-id> |
%(global_request_id)s |
The id of the request that started the chain in another service, when the caller passed one. None when the request started here. |
req-<uuid> |
%(request_id)s |
The id of the request in this service, generated by oslo.middleware as req-<uuid4> for each API call and returned to the client in the x-openstack-request-id response header. |
<user-id> ... <project-domain-id> |
%(user_identity)s |
Who made the request: logging_user_identity_format, default %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s. These are ids, not names; a missing value prints -. |
[instance: <uuid>] |
%(instance)s |
Present when the code passed the instance (Nova does so for most compute actions): instance_format / instance_uuid_format, default [instance: %(uuid)s]. Empty otherwise. This is what makes grep <instance-uuid> work across Nova's logs. |
<message> |
%(message)s |
The message. When the line carries an exception, oslo.log appends : <error summary> (the exception's type and text) if the format does not place it elsewhere. |
Three details that matter when you grep:
[-]means "no request". Lines without a request id use the default format, with a literal[-]. Periodic tasks (resource tracker updates, power state sync, agent reports) log this way: they belong to no API call, so you will not find them by request id. Find them by time and by module instead.-
A traceback is many lines, each with a prefix. When an exception is logged, every line of the traceback is printed with
logging_exception_prefix, default%(asctime)s.%(msecs)03d %(process)d ERROR %(name)s %(instance)s. Those lines carry no request id. To read a traceback that belongs to a request, find the request'sERRORline, then read the following lines from the same pid and timestamp:grep -n "<request-id>" /var/log/kolla/nova/nova-compute.log | grep ERROR # line numbers sed -n '<line>,+40p' /var/log/kolla/nova/nova-compute.log # the traceback after it -
DEBUG lines are longer. At debug level,
oslo.logappendslogging_debug_format_suffix, default%(funcName)s %(pathname)s:%(lineno)d: the function, file and line that logged. Useful to read the code that produced the line; noise otherwise.
Request id and global request id¶
The two ids answer different questions
(oslo_middleware/request_id.py,
oslo_context/context.py):
- Each API service generates its own
req-<uuid>for every request it receives, and returns it in thex-openstack-request-idheader. - When a service calls another one, it sends an
X-Openstack-Request-Idheader with its global id: its own global request id if it has one, otherwise its request id (RequestContext.global_id). The receiving service accepts it only if it has the exactreq-<uuid>shape, and stores it as itsglobal_request_id.
So, for one openstack server create:
nova-apilogs[None req-A ...]: the request started in Nova;- the Neutron calls that Nova makes for this instance appear in
neutron-server.logas[req-A req-N ...]: the global id is Nova's, the local one is Neutron's.
To follow one user action across services, grep for the id from the first service (req-A): it appears as the
request id there and as the global request id in the services it called. Whether a given call carries the global id
depends on the calling code passing its context; check it on your release before you rely on it for a given hop.
The tracing method itself is on the
tracing page.
Log levels¶
oslo.log uses the Python levels, plus two of its own
(oslo_log/handlers.py,
oslo_log/log.py):
| Level | Value | How to read it |
|---|---|---|
CRITICAL |
50 | The service is about to stop, or cannot start. Read the lines just before it. |
ERROR |
40 | An operation failed. Not always the cause: a failure in a dependency (database, bus, another API) shows as an ERROR in every service that used it. Find the first one in time. |
WARNING |
30 | Something unexpected that the service worked around: a retry, a timeout that recovered, a deprecated option. A burst of warnings often precedes an error. |
INFO |
20 | The normal story: requests served, instances spawned, agents reporting. Default level. |
DEBUG |
10 | Each step, payloads, RPC calls. Off by default. |
TRACE |
5 | oslo.log defines it; rarely used by the services. |
The default level is INFO (debug = false). Some noisy libraries are pinned higher whatever the service level,
by default_log_levels: for example amqp=WARN, sqlalchemy=WARN, keystonemiddleware=WARN,
oslo.messaging=INFO, urllib3.connectionpool=WARN. That is why debug = true on a service does not show the SQL
it runs or every keystonemiddleware step; to see one of those, change its entry in default_log_levels for that
service.
Turning debug on for one service, then off¶
Debug on a busy controller writes a lot, quickly, and can fill the disk that holds the logs. Turn it on for one service on one host, for the minutes you need, and turn it off.
Kolla-Ansible: each container reads its configuration from /etc/kolla/<container-config-dir>/ on the host,
copied into the container at start (config_strategy: COPY_ALWAYS by default). Kolla-Ansible writes
debug = {{ <service>_logging_debug }} in the [DEFAULT] section, which is False unless
openstack_logging_debug or <service>_logging_debug is set in globals.yml.
# On the host that runs the service (example: nova-compute on one compute)
sudo grep -n '^debug' /etc/kolla/nova-compute/nova.conf # expect debug = False; the first match is in [DEFAULT]
sudo df -h "$(sudo readlink -f /var/log/kolla)" # room for the extra log volume (Docker or Podman)
⚠️ State-changing. Edits the service's configuration on this host and restarts its container. The restart can take about a minute (62 s on the lab), because the stop waits up to
docker_graceful_timeout(60 s by default); the service is unavailable meanwhile: requests in flight to it are lost or retried, and fornova_computethe instances keep running (see nova-compute down, instances still running). Do it on one host only. Undo with the commands below, or with the nextkolla-ansible reconfigure, which rewrites the file from its template.
# first match only: the one in [DEFAULT] (nova.conf also has debug in [glance] and [guestfs])
sudo sed -i '0,/^debug = False/s//debug = True/' /etc/kolla/nova-compute/nova.conf
sudo systemctl restart kolla-nova_compute-container.service
sudo grep -c ' DEBUG ' /var/log/kolla/nova/nova-compute.log # the count grows
Back off, as soon as you have what you need:
⚠️ State-changing. Same restart as above.
sudo sed -i '0,/^debug = True/s//debug = False/' /etc/kolla/nova-compute/nova.conf
sudo systemctl restart kolla-nova_compute-container.service
To keep debug for a whole service across all hosts, set <service>_logging_debug: "True" (for example
nova_logging_debug) in globals.yml and run kolla-ansible reconfigure for that service: that restarts the
service everywhere, and needs the same reconfigure to undo.
Any: set debug = true in the [DEFAULT] section of the service's configuration file, restart the unit, and
undo the same way.
⚠️ State-changing. Restarts the service on this host; same impact as above.
crudini --set /etc/nova/nova.conf DEFAULT debug true # or edit the file by hand
systemctl restart <unit>
debug is declared mutable in oslo.log, so services that reload mutable options on SIGHUP (through
oslo.service) can pick it up without a restart. What else a SIGHUP does is service-specific (for example,
nova-scheduler refreshes its cell cache, per Nova's
cells documentation); this page sticks to the restart,
whose effects are known.
Guru Meditation report: the state of a live process¶
When a service hangs without logging anything, its log cannot tell you why. Nova and most OpenStack services can
dump a Guru Meditation report (GMR): version, configuration, the stack of every thread and green thread. Sending
SIGUSR2 to the process writes it to the process's standard error, or to a file in [oslo_reports] log_dir when set
(Nova GMR reference). Services run under uWSGI intercept
SIGUSR2: for them, configure [oslo_reports] file_event_handler and touch that file (not run on the lab).
Any:
sudo kill -USR2 <pid-of-the-service>
On Kolla-Ansible, standard error of the service is the container's output, so the report appears in
docker logs <container> and in the journal of its unit (journalctl -u kolla-<container>-container.service).
On a lab, sudo kill -USR2 on the nova-compute process (its host PID from sudo docker top nova_compute) wrote a
report of more than 2000 lines there at once, starting with ==== Guru Meditation ====, then Package,
Threads, ...; the service kept running.
Logs that do not use oslo.log¶
The infrastructure around OpenStack has its own formats. Kolla-Ansible's fluentd parsers describe them; each has its own page in this folder (to come).
| Log | Line starts with | Source of the format |
|---|---|---|
| RabbitMQ | <date> <time>.<ms>[<digits><+HH:MM>] [<level>] <message> |
fluentd input/ |
| MariaDB / Galera | a date, YYYY-MM-DD or YYMMDD (the server and its mysqld_safe wrapper use different formats) |
fluentd input/ |
| Open vSwitch, OVN | <date>T<time>.<ms>Z|<sequence>|<module>|<level>|<message> (UTC) |
fluentd input/ |
| libvirtd | <date> <time>.<ms><offset>: <pid>: <level> : <message> |
fluentd input/ |
| uWSGI request lines | [pid: <pid>|app: <n>|req: <n>/ |
fluentd input/ |
Two of these are in UTC (Open vSwitch and OVN, the trailing Z), while oslo.log lines are in the host's local
time without a zone. On a host whose time zone is not UTC, the same event appears with different hours in
ovs-vswitchd.log and in neutron-openvswitch-agent.log.
Validity and limits¶
-
Written from the 2026.1 sources of
oslo.log,oslo.context,oslo.middlewareand Kolla-Ansible, and from the Nova and Kolla-Ansible documentation linked above. Checked on an all-in-one Kolla-Ansible 2026.1 lab (ML2/OVS, no HAProxy, host in UTC) on 2026-10-27: the log directories and file names of the table (plusnova-metadata.log,nova-novncproxy.log,privsep-helper.log,dnsmasq.log,mariadb-bootstrap.log,ansible.log), the commands of Where the logs are (withsudo), debug on and off fornova_compute(systemctl restartreturned after 62 s), and the GMR. Real lines seen there, redacted:2026-10-27 13:15:17.272 7 INFO nova.compute.manager [-] [instance: <server-id>] VM Stopped (Lifecycle Event) [pid: 27|app: 0|req: 175/696] 192.0.2.10 () {28 vars in 305 bytes} [Tue Oct 27 13:20:12 2026] GET / => generated 377 bytes in 3 msecs (HTTP/1.1 200) 3 headers in 91 bytes (1 switches on core 0) 2026-10-27T13:15:46.733Z|00844|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: 8 flow_mods in the 4 s starting 42 s ago ... 2026-10-27 13:19:25.681480+00:00 [info] <0.395310.0> connection 192.0.2.10:35334 -> 192.0.2.10:5672 - nova-compute ... 2026-10-27 13:19:16.682+0000: 5159: error : virQEMUCapsCacheLookupDefault:5959 : invalid argument: ... 2026-10-27 13:09:40 5952 [Warning] Aborted connection 5952 to db: 'heat' user: 'heat' host: ...(
nova-compute.log,nova-api-uwsgi.log,ovs-vswitchd.log,rabbit@<host>.log,libvirtd.log,mariadb.log.) The RabbitMQ and libvirtd lines carry their offset (+00:00,+0000), the MariaDB line does not. -
File names are those of Kolla-Ansible 2026.1 defaults. Earlier releases ran some APIs differently (no
-uwsgi.logfile), and a deployment can overridelog_file, the format strings, or the container names. - For package installs, only the Nova file names and unit names come from upstream documentation; the other projects' paths depend on the distribution's packaging.
- The global request id is set only on the calls where the calling code passes its context; this page does not list which inter-service calls do.
- The HAProxy file name (
haproxy_latest) comes from thefluentdrole defaults; the exact file name on disk (with fluentd's file output suffix) was not checked. - The GMR was checked for
nova-computeonly, not for a service under uWSGI. - Each instance's QEMU log is inside the
nova_libvirtcontainer:sudo docker exec nova_libvirt tail /var/log/libvirt/qemu/<instance-name>.logworked on the lab.
Related¶
- What a VM creation traverses, and how to trace one failed boot by request id
- The logs of one compute do not line up with the controllers: clock skew
- Logs index
- oslo.log configuration options: https://docs.openstack.org/oslo.log/latest/configuration/index.html
- Kolla-Ansible central logging guide: https://docs.openstack.org/kolla-ansible/latest/reference/logging-and-monitoring/central-logging-guide.html
Provenance: written by agent:ozarland-author, from upstream source code and documentation (2026.1). Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your deployment method, and what happened.