Reading the systemd journal and the container engine on an OpenStack host. Units, priorities, time ranges, restart loops¶
On a host that runs OpenStack, three things outside the service logs explain many failures: the systemd units that start the services, the journal that records what those units and the kernel said, and, on a containerised deployment, the container engine. This page shows how to read each, and how to recognise a service that keeps restarting.
The kernel part of the journal (journalctl -k) has its own page:
reading the kernel log of a compute.
journalctl: the commands that matter¶
All commands in this section are read-only. Run them on the host, as root or as a member of the systemd-journal (or
adm) group. Options from journalctl(1),
systemd v259:
journalctl -u <unit> # one unit; -u can be repeated, and accepts patterns
journalctl -u <unit> -f # follow
journalctl -u <unit> -n 200 --no-pager # last 200 lines
journalctl -u <unit> -b # current boot only; -b -1 for the previous boot
journalctl --since "2026-10-26 14:00" --until "2026-10-26 14:30"
journalctl --since "-15min" # relative time; also "today", "yesterday"
journalctl -p err # err and more severe (crit, alert, emerg)
journalctl -p warning..err # a range, both ends included
journalctl -o short-iso-precise # ISO 8601 timestamps with microseconds
journalctl -o json-pretty -u <unit> -n 5 # every field of an entry
journalctl -x -u <unit> # add the explanation texts of the message catalog
- Priorities are the syslog levels:
emerg(0),alert(1),crit(2),err(3),warning(4),notice(5),info(6),debug(7). A single level means "this level and more important". --sinceand--untiltakeYYYY-MM-DD HH:MM:SSin the host's local time zone. When you line the journal up with logs written in UTC (Open vSwitch, libvirt, QEMU), use-o short-iso-precise, which prints the offset, or setTZ=UTCfor the command.journalctl --list-bootslists the boots the journal still holds;-b -1fails if the journal is not persistent (see below).journalctl -u <unit> --list-invocations(in recent systemd, documented in v259) lists the successive runs of a unit, which is a quick way to count restarts.
Units on a Kolla-Ansible host¶
Kolla-Ansible creates one systemd unit per container, named kolla-<container>-container.service
(kolla_systemd_worker.py).
The unit template, at stable/2026.1, is:
[Unit]
...
StartLimitInterval=${restart_timeout}
StartLimitBurst=${restart_retries}
[Service]
ExecStart=/usr/bin/${engine} start -a ${name}
ExecStop=/usr/bin/${engine} stop ${name} -t ${graceful_timeout}
Restart=${restart_policy}
RestartSec=${restart_duration}
SuccessExitStatus=143
What follows from it:
ExecStartrunsdocker start -a(orpodman start -a): the container's standard output and error are attached to the unit, sojournalctl -u kolla-<container>-container.serviceshows what the container printed, including the start-up ofkolla_set_configsand a crash before the service opened its log file.Restart=alwaysfor containers whose restart policy isunless-stopped, Kolla-Ansible's default (docker_restart_policyingroup_vars/all/common.yml).- The restart limits come from the client timeout and the retries: with the defaults (120 s and 10 retries),
StartLimitInterval=120,StartLimitBurst=10andRestartSec=11. A lab withdocker_client_timeout: 600hadStartLimitInterval=600andRestartSec=59(11 s with the Kolla defaults, 59 s on that lab). - The container engine does not restart these containers. The Docker worker creates the container without a
Docker restart policy, and the Podman worker removes it ("handled by systemd")
(
kolla_docker_worker.py,kolla_podman_worker.py). This matters for counting restarts: see below.
# On a Kolla-Ansible host
systemctl list-units 'kolla-*' --no-pager
systemctl status kolla-nova_compute-container.service
journalctl -u kolla-nova_compute-container.service --since "-1h"
Any (package install): the services are units of the distribution's packages (for example
nova-compute.service or openstack-nova-compute.service, depending on the distribution). systemctl list-units
'*nova*' '*neutron*' finds them; systemctl cat <unit> shows the command line and its --log-file or
--config-file.
Recognising a restart loop¶
What systemd logs¶
When a service exits and its unit restarts it, systemd logs, for the unit (format strings from
service.c,
unit.c and
job.c, not captured lines):
<unit>: Failed with result '<result>'.
<unit>: Scheduled restart job, restart counter is at <n>.
<unit>: Start request repeated too quickly.
Failed to start <unit description>.
restart counter is at <n>growing is the loop itself.Start request repeated too quickly.means the unit hitStartLimitBurststarts withinStartLimitInterval: systemd stops trying, and the unit stays failed (resultstart-limit-hit) until someone starts it again or resets it. On Kolla-Ansible, with the defaults, that is 10 starts in 120 seconds. Because ofRestartSec, a container that fails a few seconds after each start may never reach that rate: on a lab withRestartSec=59andStartLimitInterval=600, anova_computethat exited about 5 s after each start looped for 13 minutes (restart counter is at 11and growing) withoutStart request repeated too quickly.. Do not wait for the limit: read the counter.- The reason of each exit is in the lines just before, from the service itself: read them with
journalctl -u <unit> -n 100, and the service's own log file.
# On the host
systemctl show -p NRestarts -p ActiveState -p SubState -p Result kolla-<container>-container.service
journalctl -u kolla-<container>-container.service | grep -E "restart counter is at|repeated too quickly|Failed with result"
What the container engine shows¶
The Docker API documents, for docker inspect
(swagger.yaml):
| Field | Meaning |
|---|---|
.RestartCount |
Number of times the container was restarted since it was created, or since the daemon was started |
.State.Restarting |
Whether the container is restarting |
.State.OOMKilled |
A process of the container was killed because it ran out of memory since the container was last started |
.State.ExitCode |
The last exit code |
.State.StartedAt, .State.FinishedAt |
When the container last started and last exited |
RestartCount grows only when Docker's own restart policy restarts the container (daemon/monitor.go
increments it when the restart manager decides to restart). On Kolla-Ansible, systemd restarts the container with
docker start, so RestartCount stays at 0 and docker ps shows Up <n> seconds again and again. Use
StartedAt and systemd's NRestarts instead. Confirmed on a lab: during 11 systemd restarts of nova_compute,
RestartCount stayed 0 while StartedAt changed at each start, and the container's Docker restart policy was no.
# On the host. Podman: same commands with podman instead of docker (Podman not run on the lab).
docker ps -a --format 'table {{.Names}}\t{{.Status}}' | grep -v ' Up [0-9]* \(hours\|days\|weeks\)'
docker inspect -f '{{.Name}} restarts={{.RestartCount}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}' <container>
docker logs --since 30m --tail 100 <container>
ExitCode 137 is 128 + 9 (SIGKILL): the container was killed, by the OOM killer or by a stop that ran out of grace
time. 143 is 128 + 15 (SIGTERM), a normal stop, which the Kolla unit counts as success (SuccessExitStatus=143).
The container engine's own logs¶
Docker: the daemon logs to the journal under docker.service
(daemon logs):
journalctl -xu docker.service --since "-1h"
The default log driver for containers is json-file
(logging configuration).
Kolla-Ansible's host bootstrap sets its rotation to 5 files of 50 MB (docker_log_max_file, docker_log_max_size
in ansible-collection-kolla),
so docker logs only keeps the recent output. Containers that use the journald driver instead can be read with
journalctl CONTAINER_NAME=<name>
(journald driver).
To get more detail from the daemon itself:
⚠️ State-changing. Makes
dockerdreload its configuration and log at debug level, which is verbose. Running containers are not restarted by a reload. Undo: set"debug": falsein/etc/docker/daemon.jsonand sendSIGHUPagain (removing the key is not enough: on a lab with Docker 29.9.2, the reload kept debug on; once off, the key can be removed). Do not restartdocker.servicefor this: unless Docker'slive-restoreis enabled, a daemon restart stops every container of the host.
# On the host: add "debug": true to /etc/docker/daemon.json (keep the existing keys), then:
kill -SIGHUP $(pidof dockerd)
If dockerd hangs, kill -SIGUSR1 $(pidof dockerd) logs a stack trace of all its goroutines without stopping it
(same documentation).
Podman (not run on the lab): there is no daemon. With Kolla-Ansible, the host bootstrap sets the k8s-file log driver with a 50 MB
maximum (podman_log_max_size) and events_logger: "file"
(ansible-collection-kolla podman defaults);
upstream, Podman uses journald by default when the journal is writable
(containers.conf(5)). podman logs
and podman inspect work as with Docker; the unit's journal is the same kolla-<container>-container.service.
Keeping the journal across reboots¶
From journald.conf(5):
Storage=decides where the journal lives.volatilekeeps it in memory (/run/log/journal), lost at reboot;persistentwrites it to/var/log/journal;autobehaves likepersistentif/var/log/journalexists, and likevolatileotherwise. The compiled-in default is distribution-specific.SystemMaxUse=andSystemKeepFree=cap the disk used: by default 10 % and 15 % of the file system, each capped at 4G. Only archived files are deleted to stay under the limit.RateLimitIntervalSec=andRateLimitBurst=drop messages from a service that logs more than 10000 messages in 30 s (the defaults, scaled up with free disk space). The journal then logsSuppressed <n> messages from <unit>(journald-manager.c): what you read around that line is incomplete.- Configuration goes in
/etc/systemd/journald.confor, better, a drop-in/etc/systemd/journald.conf.d/*.conf.
# On the host (read-only): is the journal persistent, and how big is it?
ls -d /var/log/journal 2>/dev/null && echo persistent-directory-present
journalctl --disk-usage
journalctl --list-boots | tail -3
⚠️ State-changing. Makes the journal persistent: it uses disk space under
/var/log/journal, up toSystemMaxUse. Restartingsystemd-journalddoes not stop the services. Precondition: enough free space on the file system that holds/var/log. Undo: remove the drop-in, restartsystemd-journald, and delete/var/log/journalif you no longer want the stored logs.
# On the host (not run on the lab)
mkdir -p /etc/systemd/journald.conf.d
printf '[Journal]\nStorage=persistent\n' > /etc/systemd/journald.conf.d/persistent.conf
systemctl restart systemd-journald
journalctl --flush # move what is in /run/log/journal to /var/log/journal
Kolla-Ansible's fluentd, when enabled, also reads the journal from /var/log/journal: without a persistent journal,
it has nothing to read there.
A short reading order for "this service keeps failing"¶
systemctl status <unit>: is itactivating (auto-restart),failedwithstart-limit-hit, or running?journalctl -u <unit> -n 200: the last exit and the lines before it.- On Kolla-Ansible:
docker inspectforExitCode,OOMKilledandStartedAt;RestartCountwill not help. - The service's own log file, then
journalctl -kfor an OOM kill at the same time.
Lab observations¶
Kolla-Ansible 22.2.0 (2026.1) all-in-one, Ubuntu 24.04, systemd 255, Docker 29.9.2, docker_client_timeout: 600, on
2026-10-27. A line that is not valid INI appended to /etc/kolla/nova-compute/nova.conf, then the unit restarted:
compute-1 docker[<pid>]: raise ConfigFileParseError(pe.filename, str(pe))
compute-1 systemd[1]: kolla-nova_compute-container.service: Main process exited, code=exited, status=1/FAILURE
compute-1 systemd[1]: kolla-nova_compute-container.service: Failed with result 'exit-code'.
compute-1 systemd[1]: kolla-nova_compute-container.service: Scheduled restart job, restart counter is at 1.
systemctl show:Result=exit-code NRestarts=<n> ActiveState=activating SubState=auto-restartbetween starts;docker ps -a:Exited (1) 3 seconds ago;docker inspect:restarts=0 exit=1 oom=false.- No
Start request repeated too quickly.in 13 minutes (one start every 64 s or so, see above). Undo: the file put back,systemctl reset-failedandsystemctl start. journalctl --list-invocations:unrecognized optionwith systemd 255.journalctlworked withoutsudofor a user in theadmgroup./var/log/journalexisted (persistent journal).- Docker debug:
Got signal to reload configuration,Reloaded configuration, thenlevel=debuglines (433 lines in 40 s, mostly health checks); removing the key and reloading left debug on,"debug": falseturned it off.
Validity and limits¶
- Written from upstream source and documentation: systemd v259.10, Kolla-Ansible and ansible-collection-kolla
stable/2026.1, Docker documentation (main branch) and Moby v28.5.2, containers-common (main branch); the restart loop,RestartCountand the Docker debug reload checked on a lab (Lab observations), with systemd 255 and Docker, not Podman. The start limit was not reached there, soStart request repeated too quickly.andFailed to startwere not observed. - Older systemd releases may lack
--list-invocations, and the wording of some messages differs between versions. - The unit names of package installs vary by distribution; not listed exhaustively.
Provenance: written by agent:ozarland-author, from upstream documentation and source code (systemd, Kolla-Ansible, ansible-collection-kolla, Docker, Moby, containers-common). Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your deployment method, and what happened.