Apply common changes (fluentd, cron, kolla_toolbox)¶
Impact Restarts on drift
How to push changes of the common role (fluentd, cron, kolla_toolbox) to hosts. In 2024.1 common is still a play
of its own, tagged common, so -t common reconfigure reconfigures every host of the common group. New in 2024.1,
the role's config step first makes sure the fluentd image is on each host and reads its labels.
The page uses the definition of drift of the Operations hub.
Note
Key answers
-t common reconfigurereconfigures every common host CODE (§2).commonreaches a host only through the "Apply role common" play: a run without tags, or a run whose tags includecommon(§3).-t novaor-t neutrondo not bring it along.- New in 2024.1: the config step can pull the fluentd image CODE. With
enable_fluentdon, every run of the role,genconfigincluded, runsensure_imageon the fluentd image and reads its labels to choose the fluentd command and user (§3).
Warning
R2: fleet-wide restarts after an enable_* change — see "Hidden coupling" in
§3. In 2024.1, enable_redis, enable_letsencrypt and enable_ovn join
the flags that change the common configs.
1. Commands¶
1.1 Apply common changes (fluentd, cron, kolla_toolbox)¶
# 2024.1: common is its own play, tagged common (site.yml:88-103)
kolla-ansible -i multinode -t common reconfigure
kolla-ansible -i multinode -t common reconfigure --limit 'compute[0:49]'
- The play targets the groups
cron,fluentd,kolla-logsandkolla-toolbox, all children ofcommon(control, network, compute, storage, monitoring) in the shipped inventory CODE (site.yml:88-103;inventory/multinode:49-54,:203-213). It runs withserial: kolla_serial(site.yml:95), soANSIBLE_SERIALbatches it (tools/kolla-ansible:531). New in 2024.1, the play also setsmax_fail_percentagefromcommon_max_fail_percentage, thenkolla_max_fail_percentage, then 100 (site.yml:96-99); with the default 100 it never stops the play early (Controlled batches). - The
chronytag used for this in Ussuri does not exist: there is nochronyrole or play in 2024.1 (noroles/chrony, nochronyinsite.yml). - Registry: with
enable_fluentdon (defaultyes,group_vars/all.yml:765), the run needs the fluentd image of the configured tag on each host of the play, or the registry to pull it (roles/common/tasks/config.yml:42-51).
2. The tag does nothing¶
Still not true in 2024.1. roles/common/tasks/main.yml:1-2 is a bare include_tasks: "{{ kolla_action }}.yml" with
no tag and no common_run guard; the tag common is set on the play (site.yml:100-101). The tasks of a dynamic
include keep the include's parent block (ansible-core 2.16.20 lib/ansible/plugins/strategy/__init__.py:831-842), a
block extends its tags with those of its play (lib/ansible/playbook/block.py:352-361; tags is an extend=True
field, lib/ansible/playbook/taggable.py:42), and the included block is then filtered by tag
(lib/ansible/plugins/strategy/linear.py:272). The included reconfigure.yml tasks therefore carry common and run
CODE. Tag semantics: Tags §2.1.
3. How common actually reaches hosts¶
-
Where it runs: the
commonrole (fluentd, cron, kolla_toolbox) runs only in the "Apply role common" play, on the hosts ofcron,fluentd,kolla-logsandkolla-toolboxCODE (site.yml:88-103). No role declarescommonas a dependency (noroles/*/metadirectory in 2024.1), and there is nocommon_runguard.Command Hosts that get commonreconfigure(no tag)all hosts of the commongroup, in the common play (the first play after the prechecks play,site.yml:77-103)-t novanone: the nova plays ( nova.yml:103-150) do not include thecommonrole CODE-t neutronnone (same reason) -t chronynone: no play carries the tag in 2024.1, only the always-tagged plays (fact gathering, host grouping) run-t haproxynone: the loadbalancer play ( site.yml:105-326) does not includecommon-t commonall hosts of the commongroup (recommended way to pushcommonchanges) -
What it can change:
- new in 2024.1, before any file: the fluentd image.
Ensure fluentd image is present for label checkrunskolla_containerensure_image, which pulls the image only when it is absent (roles/common/tasks/config.yml:42-51;module_utils/kolla_docker_worker.py:501-503,module_utils/kolla_podman_worker.py:675-677). Its condition isenable_fluentdalone, so it runs on every host of the play. The labelsfluentd_binaryandfluentd_userare then read withdocker_image_infoorpodman_image_infoand set the factsfluentd_cmdandfluentd_user,td-agentwhen a label is missing (config.yml:53-91). The 2024.1 kolla image sets both labels tofluentd(kolladocker/fluentd/Dockerfile.j2:3,6); config.jsonof the three containers (config.yml:93-102). fluentd's command is now<fluentd_cmd> -c /etc/<fluentd_cmd>/<fluentd_cmd>.conf -o /var/log/kolla/fluentd/fluentd.log, andtd-agent.confis copied to that path (roles/common/templates/fluentd.json.j2:1-12). kolla_toolbox's command is nowkolla_toolboxinstead ofsleep infinity(roles/common/templates/kolla-toolbox.json.j2:2);td-agent.conf(fluentd inputs/outputs). It is built fromfluentd_input_openstack_services(theenable_*flags of the OpenStack services,roles/common/defaults/main.yml:154-210) read byconf/input/00-global.conf.j2:2-5; the input files enabled byenable_nova/enable_nova_libvirt_container,enable_prometheus_fluentd_integration, and, new in 2024.1,enable_letsencryptandenable_fluentd_systemd(the journal) (config.yml:149-169); the OVN log inputs whenenable_ovn, on the OVN hosts only (roles/common/templates/conf/input/10-openvswitch.conf.j2:31-67); the local syslog outputs ofsyslog_facilities(swift and haproxy, and the glance and neutron TLS proxies, each only on its own hosts,roles/common/defaults/main.yml:82-104); the Elasticsearch (elasticsearch_addressdefined) and OpenSearch (enable_opensearchor an externalopensearch_address) outputs, now withbulk_message_request_thresholdandchunk_limit_size(roles/common/defaults/main.yml:147,152); and custom files under{{ node_custom_config }}/fluentd/CODE (roles/common/tasks/config.yml:104-203);logrotate.conf(cron, built fromcron_logrotate_services, task atroles/common/tasks/config.yml:205-270);- the RabbitMQ Erlang cookie,
rabbitmq-env.confanderl_inetrcin kolla_toolbox (only withenable_rabbitmq,config.yml:272-323); - with Podman only,
/etc/tmpfiles.d/kolla.confon the host, followed bysystemd-tmpfiles --create(config.yml:30-40); - container definitions of fluentd, cron and kolla_toolbox (image tag, volumes, environment, dimensions). New in
2024.1, fluentd always mounts
/var/log/journal:/var/log/journal:ro(roles/common/defaults/main.yml:124), and kolla_toolbox mounts/runwith:sharedonly under Docker, plusrun_default_volumes_<engine>(roles/common/defaults/main.yml:22,111;group_vars/all.yml:165-172).
- new in 2024.1, before any file: the fluentd image.
- What restarts: any drift restarts all three containers on that host (loop-notify in
check-containers.yml) CODE (roles/common/tasks/check-containers.yml:2-17,compare_containerwith a loopednotify: "Restart {{ item.key }} container"; ansible-core 2.16.20lib/ansible/executor/task_executor.py:124-130,lib/ansible/plugins/strategy/__init__.py:648-673; Tags §2.4). Theconfig.jsontask is looped with the samenotify(roles/common/tasks/config.yml:93-102).- First 2024.1 run after 2023.1: a container that no 2024.1 run has recreated or restarted yet is reported as
changed by its
unit file (F16); for fluentd, the new journal volume also makes the container
differ (
roles/common/defaults/main.yml:124), so it is recreated CODE. - A change of
config.json'scommandalone (e.g. kolla_toolbox's) is not seen bykolla_set_configs --check(Kolla images §3.2). A change of the fluentd command from the labels also moves the destination oftd-agent.conf(fluentd.json.j2:8-9), which the check reports as a missing destination (Kolla images §3.3) CODE.
- First 2024.1 run after 2023.1: a container that no 2024.1 run has recreated or restarted yet is reported as
changed by its
unit file (F16); for fluentd, the new journal volume also makes the container
differ (
- Impact:
- fluentd: log shipping gap. The Elasticsearch and OpenSearch outputs buffer to files under
/var/lib/fluentd/data, thefluentd_datavolume, which a restart or a recreate keeps (roles/common/defaults/main.yml:123;roles/common/templates/conf/output/01-es.conf.j2:28-33,03-opensearch.conf.j2:28-33). The tail read positions are in/var/run/<fluentd_user>/,/var/run/fluentd/with the 2024.1 image, on no volume (roles/common/templates/conf/input/00-global.conf.j2:12; kolladocker/fluentd/Dockerfile.j2:41-42): NOT AVAILABLE whether they survive a config-only restart and whether they are lost when the container is recreated (e.g. an image or volume change) (Kolla images §5). - cron: log rotation is delayed.
- kolla_toolbox: no service template references it (only
roles/common/templates/kolla-toolbox.json.j2names it underroles/*/templates); Kolla's own tasks use it through thekolla_toolboxmodule, which fails when the container is not running (library/kolla_toolbox.py:150-153,:206-216) CODE. Its new command exits 0 on TERM or QUIT (kolladocker/kolla-toolbox/kolla_toolbox.sh:3-16). - VM and API impact: NOT AVAILABLE. None of the three is a container that Tags §3.1 traces to running VMs CODE.
- fluentd: log shipping gap. The Elasticsearch and OpenSearch outputs buffer to files under
-
Hidden coupling: enabling a service that appears in either list (most OpenStack services, e.g. barbican) changes
td-agent.confand/orlogrotate.confon every host (Repository architecture §5.1). New in 2024.1 CODE:enable_redischangeslogrotate.confon every cron host (roles/common/tasks/config.yml:254). With Octavia and the amphora driver, the jobboard is on by default (group_vars/all.yml:832) and the Octavia precheck requiresenable_redis(roles/octavia/tasks/precheck.yml:67-72), so enabling Octavia can bring this change along;enable_letsencryptadds an input totd-agent.confon every fluentd host (config.yml:166-167); its logrotate template is not in the logrotate list (config.yml:212-261), sologrotate.confdoes not change;enable_ovnchangestd-agent.confonly on the hosts of the OVN groups (roles/common/templates/conf/input/10-openvswitch.conf.j2:31-67).
The common containers restart on every common host whose file changed at the next run that includes the common play: any run without tags, or a run with
-t common. A tagged run such as-t novadoes not trigger it (Reconfigure Nova §3).
4. Sequence, risks and verification¶
- Sequence on each reached host, from
roles/common/tasks/deploy.yml:2-9(reconfigure.ymlimports it,roles/common/tasks/reconfigure.yml:2):- config (
roles/common/tasks/config.yml:2-323): config directories; CA certificates ifkolla_copy_ca_into_containers; with Podman, the tmpfiles config; new:ensure_imageof the fluentd image, then its labels and thefluentd_cmd/fluentd_userfacts;config.json×3; custom fluentd files (found once, on the deploy host,run_once+delegate_to: localhost);td-agent.conf; cronlogrotate.conf; RabbitMQ Erlang cookie,rabbitmq-env.confanderl_inetrcfor kolla_toolbox.rabbitmq-env.confanderl_inetrcnotify no handler (config.yml:297-323); a change to them is found by the next step, since both are listed in kolla_toolbox'sconfig.json(roles/common/templates/kolla-toolbox.json.j2:10-21); - check-containers: one looped
compare_containerover fluentd, kolla_toolbox and cron; - bootstrap:
kolla_logsvolume and the/var/log/kollalink, now tocontainer_engine_volumes_path(roles/common/tasks/bootstrap.yml:1-16;group_vars/all.yml:156-158); - flush handlers, in definition order: Restart fluentd → Restart kolla-toolbox (→ toolbox
ansible --versioninit, withkolla_container_engine exec) → Restart cron (roles/common/handlers/main.yml:2,:18,:37-40,:42).
- config (
genconfig(kolla_action=config,tools/kolla-ansible:559) runsconfig.ymlonly: no restart, since every restart handler carrieswhen: kolla_action != "config"(roles/common/handlers/main.yml:15-16), but it does runensure_imageand can pull the fluentd image CODE.- Hosts, parallelism: all reached hosts at once, unless batched (
serial: kolla_serial,site.yml:95). -
Interruption: one stop/start per container. With Docker, a restart goes through the container's systemd unit and a recreate stops it through the unit when one exists; the unit's
ExecStopis<engine> stop <name> -t <graceful_timeout>, withdocker_graceful_timeout: 60(module_utils/kolla_docker_worker.py:415-431,:439-459;module_utils/kolla_systemd_worker.py:32;group_vars/all.yml:139,151; Repository architecture §6). A recreate pulls the image first when it is absent (module_utils/kolla_docker_worker.py:341-363). Log shipping pauses, cron rotation is delayed, and Kolla's ownkolla_toolboxtasks fail while the container is not running. NOT AVAILABLE Whether VMs, APIs, networking or storage are affected (§3). -
Risks: R2 (fleet-wide restarts after an
enable_*change), R25 (-t commonmistaken for a reconfigure; in 2024.1-t commondoes reconfigure, see §2), F16 (first 2024.1 check after 2023.1). - Verification:
- before: make sure the fluentd image of the configured tag is on the hosts or reachable in the registry; then
genconfig, thendocker exec -u root <c> /usr/local/bin/kolla_set_configs --check; echo $?in each offluentd,cronandkolla_toolbox(podman execwith Podman; 1 in any of them = all three will restart, by loop-notify; alogrotate.conf-only change, e.g. togglingenable_redis, shows up only incron). The check does not cover image, volume, unit-file orcommandchanges; - after:
docker ps --filter name=fluentd --filter name=cron --filter name=kolla_toolboxshows fresh start times;docker logs fluentdhas no parse errors; logs still reach the configured output.
- before: make sure the fluentd image of the configured tag is on the hosts or reachable in the registry; then
Related¶
- Risk register — the risks this page names: R2 (fleet-wide
restarts after an
enable_*change) and R25 (-t commonmistaken for a reconfigure) - Common: fluentd, cron, kolla_toolbox — service card and where to look
- Chrony — the Ussuri tag used to push
commonchanges; the role does not exist in 2024.1 - Pre-execution checklist — the checks before every production run
- Controlled batches —
genconfigfirst, then batches;max_fail_percentage - Tags §2.1 — dynamic
include_tasksand tag inheritance: the tag semantics behind §2 - Repository architecture §5.1 —
the
commonrole and thecommon_runguard (gone since 2023.1) - Why a small change creates a large workload —
enable_<new service>: yeschanges fluentd/logrotate on every host