Apply common changes (fluentd, cron, kolla_toolbox)¶
Impact Restarts on drift
How to push changes of the common role (fluentd, cron, kolla_toolbox) to hosts. In 2023.1 common is a play of its
own, tagged common, so -t common reconfigure reconfigures every host of the common group.
The page uses the definition of drift of the Operations hub.
Note
Key answers
1. Commands¶
1.1 Apply common changes (fluentd, cron, kolla_toolbox)¶
# 2023.1: common is its own play, tagged common (site.yml:86-97)
kolla-ansible -i multinode -t common reconfigure
kolla-ansible -i multinode -t common reconfigure --limit 'compute[0:49]'
- The play targets the groups
cron,fluentd,kolla-logsandkolla-toolbox, all children ofcommon(control, network, compute, storage, monitoring) in the shipped inventory CODE (site.yml:86-97;inventory/multinode:48-53, :239-249). It runs withserial: kolla_serial(site.yml:93), soANSIBLE_SERIALbatches it. - The
chronytag used for this in Ussuri no longer exists: there is nochronyrole or play in 2023.1 (noroles/chrony, nochronyinsite.yml).
2. The tag does nothing¶
No longer true in 2023.1. roles/common/tasks/main.yml:1-2 is a bare include_tasks: "{{ kolla_action }}.yml" with no
tag and no common_run guard; the tag common is set on the play (site.yml:86-97). Tasks of a dynamic include get
the include's parent block (ansible-core 2.14.18 lib/ansible/plugins/strategy/__init__.py:819-820), and a block
extends its tags with those of its play (lib/ansible/playbook/block.py:355-363; tags is an extend=True field,
lib/ansible/playbook/taggable.py:31). The included reconfigure.yml tasks therefore carry common and run CODE.
Tag semantics: Tags §2.1.
3. How common actually reaches hosts¶
-
Where it runs: the
commonrole (fluentd, cron, kolla_toolbox) runs only in the "Apply role common" play, on the hosts ofcron,fluentd,kolla-logsandkolla-toolboxCODE (site.yml:86-97). No role declarescommonas a dependency any more (noroles/*/metadirectory in 2023.1), and thecommon_runguard is gone.Command Hosts that get commonreconfigure(no tag)all hosts of the commongroup, in the common play (the first service play after prechecks,site.yml:86-97)-t novanone: the nova plays ( nova.yml:99-135) do not include thecommonrole CODE-t neutronnone (same reason) -t chronynone: no play carries the tag in 2023.1, only the always-tagged plays (fact gathering, host grouping) run-t haproxynone: the loadbalancer play ( site.yml:99-110) does not includecommon-t commonall hosts of the commongroup (recommended way to pushcommonchanges) -
What it can change:
td-agent.conf(fluentd inputs/outputs). It is built fromfluentd_input_openstack_services(theenable_*flags of the OpenStack services,roles/common/defaults/main.yml:138-202) read byconf/input/00-global.conf.j2:2-5, the input files enabled byenable_nova/enable_nova_libvirt_containerandenable_prometheus_fluentd_integration, the local syslog outputs ofsyslog_facilities(swift and haproxy, and the glance and neutron TLS proxies, each only on its own hosts,roles/common/defaults/main.yml:82-102), the Elasticsearch (elasticsearch_addressdefined) and OpenSearch (enable_opensearchor an externalopensearch_address) outputs, and custom files under{{ node_custom_config }}/fluentd/CODE (roles/common/tasks/config.yml:41-136). The fluentd image label check and the Monasca output of Ussuri are gone;logrotate.conf(cron, built fromcron_logrotate_services, task atroles/common/tasks/config.yml:138);- the RabbitMQ Erlang cookie,
rabbitmq-env.confanderl_inetrcin kolla_toolbox (only withenable_rabbitmq); - container definitions of fluentd, cron and kolla_toolbox (image tag, volumes, environment, dimensions).
- What restarts: any drift restarts all three containers on that host (loop-notify in
check-containers.yml) CODE (roles/common/tasks/check-containers.yml:2-17,compare_containerwith a loopednotify: "Restart {{ item.key }} container"; ansible-core 2.14.18lib/ansible/plugins/strategy/__init__.py:614-631; Tags §2.4). - Impact:
- fluentd: log shipping gap. The Elasticsearch and OpenSearch outputs buffer to files under
/var/lib/fluentd/data, thefluentd_datavolume, which a restart or a recreate keeps (roles/common/defaults/main.yml:123;roles/common/templates/conf/output/01-es.conf.j2:27-31,03-opensearch.conf.j2:27-31). The tail read positions are in/var/run/td-agent/, on no volume (roles/common/templates/conf/input/00-global.conf.j2:12): NOT AVAILABLE whether they survive a config-only restart and whether they are lost when the container is recreated (e.g. an image or volume change) (Kolla images §5). - cron: log rotation is delayed.
- kolla_toolbox: no service template references it (
roles/*/templates); Kolla's own tasks use it CODE. - VM and API impact: NOT AVAILABLE. None of the three is a container that Tags §3.1 traces to running VMs CODE.
- fluentd: log shipping gap. The Elasticsearch and OpenSearch outputs buffer to files under
- Hidden coupling: enabling a service that appears in either list (most OpenStack services, e.g. barbican) changes
td-agent.confand/orlogrotate.confon every host (Repository architecture §5.1). In 2023.1 the common containers restart on every common host at the next run that includes the common play: any run without tags, or a run with-t common. A tagged run such as-t novano longer triggers it (Reconfigure Nova §3).
4. Sequence, risks and verification¶
- Sequence on each reached host, from
roles/common/tasks/deploy.yml:1-9(reconfigure.ymlimports it):- config: config directories;
config.json×3; custom fluentd files;td-agent.conf; cronlogrotate.conf; RabbitMQ Erlang cookie,rabbitmq-env.confanderl_inetrcfor kolla_toolbox; - check-containers: one looped
compare_containerover fluentd, kolla_toolbox and cron; - bootstrap:
kolla_logsvolume and the/var/log/kollalink (roles/common/tasks/bootstrap.yml:1-16); - flush handlers, in definition order: Restart fluentd → Restart kolla-toolbox (→ toolbox
ansible --versioninit) → Restart cron (roles/common/handlers/main.yml:2, :18, :37-40, :42).
- config: config directories;
- Hosts, parallelism: all reached hosts at once, unless batched (
serial: kolla_serial,site.yml:93). -
Interruption: one stop/start per container. The stop goes through the container's systemd unit,
docker stop -t <graceful_timeout>, withdocker_graceful_timeout: 60(module_utils/kolla_docker_worker.py:436-452,module_utils/kolla_systemd_worker.py:33,group_vars/all.yml:140; Repository architecture §6). Log shipping pauses, cron rotation is delayed, and Kolla's ownkolla_toolboxtasks wait. NOT AVAILABLE Whether VMs, APIs, networking or storage are affected (§3). -
Risks: R2 (fleet-wide restarts after an
enable_*change), R25 (-t commonmistaken for a reconfigure; in 2023.1-t commondoes reconfigure, see §2). - Verification:
- before:
genconfig, thendocker exec -u root <c> /usr/local/bin/kolla_set_configs --check; echo $?in each offluentd,cronandkolla_toolbox(1 in any of them = all three will restart, by loop-notify; alogrotate.conf-only change, e.g. togglingenable_mariadb, shows up only incron). The check does not cover image or volume changes; - after:
docker ps --filter name=fluentd --filter name=cron --filter name=kolla_toolboxshows fresh start times;docker logs fluentdhas no parse errors; logs still reach the configured output.
- before:
Related¶
- Risk register — the risks this page names: R2 (fleet-wide
restarts after an
enable_*change) and R25 (-t commonmistaken for a reconfigure) - Common: fluentd, cron, kolla_toolbox — service card and where to look
- Chrony — the Ussuri tag used to push
commonchanges; the role no longer exists in 2023.1 - Pre-execution checklist — the checks before every production run
- Tags §2.1 — dynamic
include_tasksand tag inheritance: the tag semantics behind §2 - Repository architecture §5.1 —
the
commonrole and thecommon_runguard (gone in 2023.1) - Why a small change creates a large workload —
enable_<new service>: yeschanges fluentd/logrotate on every host