Apply common changes (fluentd, cron, kolla_toolbox)¶
Impact Restarts on drift
How to push changes of the common role (fluentd, cron, kolla_toolbox) to hosts. -t common reconfigure reconfigures
no host, so the commands use a play that reaches the hosts first, such as -t chrony reconfigure.
The page uses the definition of drift of the Operations hub.
Note
Key answers
1. Commands¶
1.1 Apply common changes (fluentd, cron, kolla_toolbox)¶
# -t common is a NO-OP in Ussuri (Tags §2.1). Use a play that reaches the hosts first:
kolla-ansible -i multinode -t chrony reconfigure # chrony play includes all hosts by default
kolla-ansible -i multinode -t chrony reconfigure --limit 'compute[0:49]'
- This also reconfigures chrony on those hosts.
- If
enable_chronyisno, use any service tag whose play covers the hosts, e.g.-t openvswitchfor network and compute hosts.
2. The tag does nothing¶
kolla-ansible -t common reconfigure executes, per play, only the include_tasks statement and the set_fact. The
included reconfigure.yml tasks are filtered out because they do not carry the common tag CODE
(roles/common/tasks/main.yml:1-10; tag semantics in Tags
§2.1). No host is reconfigured.
3. How common actually reaches hosts¶
-
Where it runs: the
commonrole (fluentd, cron, kolla_toolbox) runs on a host during the first play of the run that includes that host and whose tag matches, then never again in that run (common_run).Command Hosts that get commonreconfigure(no tag)all hosts, in the chrony play (first play; chrony is enabled by default on all physical groups) -t novaall nova hosts including every compute CODE ( nova.yml:117-135,roles/)nova-cell/ meta/ main.yml:3 -t neutronnetwork + compute + neutron-server hosts -t chronyall hosts (recommended way to push commonchanges)-t haproxyhaproxy hosts -
What it can change:
td-agent.conf(fluentd inputs/outputs). It is built from theenable_*flags of the services influentd_input_openstack_services,enable_swift/enable_haproxy(local outputs on swift and haproxy hosts only),glance_enable_tls_backend(glance-tls-proxy output on every host), the fluentd image'sfluentd_versionlabel (selects the rewrite filter, 0.12 if absent), custom files under{{ node_custom_config }}/fluentd/and the Elasticsearch/Monasca settings;logrotate.conf(cron, built fromcron_logrotate_services);- the RabbitMQ erlang cookie in kolla_toolbox;
- container definitions of fluentd, cron and kolla_toolbox (image tag, volumes).
- What restarts: any drift restarts all three containers on that host (loop-notify in
check-containers.yml) CODE (roles/common/tasks/check-containers.yml:2-17; Tags §2.4). - Impact:
- fluentd: log shipping gap. The Elasticsearch and Monasca outputs buffer to files under
/var/lib/fluentd/data, thefluentd_datavolume, which a restart or a recreate keeps (roles/common/defaults/main.yml:93;roles/common/templates/conf/output/01-es.conf.j2:26-27,02-monasca.conf.j2:13-14). The tail read positions are in/var/run/<fluentd_binary>/, on no volume (roles/common/templates/conf/input/00-global.conf.j2:14): NOT AVAILABLE whether they survive a config-only restart and whether they are lost when the container is recreated (e.g. an image or volume change) (Kolla images §5). - cron: log rotation is delayed.
- kolla_toolbox: no service template references it (
roles/*/templates); Kolla's own tasks use it CODE. - VM and API impact: NOT AVAILABLE. None of the three is a container that Tags §3.1 traces to running VMs CODE.
- fluentd: log shipping gap. The Elasticsearch and Monasca outputs buffer to files under
- Hidden coupling: enabling a service that appears in either list (most OpenStack services, e.g. barbican) changes
td-agent.confand/orlogrotate.confon every host (Repository architecture §5.1). The common containers therefore restart on whatever hosts the next run reaches, which can be a seemingly unrelated-t novarun (Reconfigure Nova §3).
4. Sequence, risks and verification¶
- Sequence on each reached host (first matching play only), from
roles/common/tasks/deploy.yml:- config: config directories; fluentd image label check;
config.json×3; custom fluentd files;td-agent.conf; cronlogrotate.conf; RabbitMQ Erlang cookie and env files for kolla_toolbox; - check-containers: one looped
compare_containerover fluentd, kolla_toolbox and cron; - bootstrap:
kolla_logsvolume and the/var/log/kollalink; - flush handlers, in definition order: Restart fluentd → Restart kolla-toolbox (→ toolbox
ansible --versioninit) → Restart cron.
- config: config directories; fluentd image label check;
- Hosts, parallelism: all reached hosts at once, unless batched.
- Interruption: one stop/start per container (stop up to 10 s, Repository architecture
§6). Log shipping
pauses, cron rotation is delayed, and Kolla's own
kolla_toolboxtasks wait. NOT AVAILABLE Whether VMs, APIs, networking or storage are affected (§3). - Risks: R2 (fleet-wide restarts after an
enable_*change), R25 (-t commonmistaken for a reconfigure). - Verification:
- before:
genconfig, thendocker exec -u root <c> /usr/local/bin/kolla_set_configs --check; echo $?in each offluentd,cronandkolla_toolbox(1 in any of them = all three will restart, by loop-notify; alogrotate.conf-only change, e.g. togglingenable_chronyorenable_mariadb, shows up only incron). The check does not cover image or volume changes; - after:
docker ps --filter name=fluentd --filter name=cron --filter name=kolla_toolboxshows fresh start times;docker logs fluentdhas no parse errors; logs still reach the configured output.
- before:
Related¶
- Risk register — the risks this page names: R2 (fleet-wide
restarts after an
enable_*change) and R25 (-t commonmistaken for a reconfigure) - Common: fluentd, cron, kolla_toolbox — service card and where to look
- Chrony — service card and where to look for the tag used to push
commonchanges - Pre-execution checklist — the checks before every production run
- Tags §2.1 — dynamic
include_tasksand tag inheritance: the tag semantics behind §2 - Repository architecture §5.1 —
the
commonrole and thecommon_runguard - Why a small change creates a large workload —
enable_<new service>: yeschanges fluentd/logrotate on every host