Ansible Automation: The Full Lifecycle

Overview

The playbooks in this section aren't independent scripts — they're stages of one system that manages a Linux host from the moment it's created to the moment something goes wrong and needs to be restored. This page ties them together and explains how they hand off to each other.

flowchart LR
    A[New LXC created] --> B[Bootstrap Playbook]
    B --> C[Host joins main inventory]
    C --> D[Patch Playbook<br/>weekly]
    C --> E[Config Backup Playbook<br/>daily]
    C --> H[Infra Patch Playbook<br/>weekly, no auto-reboot]
    D --> F{Something breaks?}
    E --> F
    H --> F
    F -->|Yes| G[Restore from backup]
    F -->|No| D
    G --> C

Stage 1 — Bootstrap (Day 0)

Ansible LXC Bootstrap

Every host's life starts here, and only here. This playbook runs exactly once per host, using temporary root/password access, and hands off to key-based automation from that point forward:

Creates the dedicated ansible service account
Installs the SSH key the rest of the automation stack will use
Applies baseline hardening (SSH config, UFW, fail2ban)
Installs common packages every host needs regardless of role

Once this playbook finishes, the host graduates from a one-off bootstrap_hosts.ini entry into the main inventory — from here on, it's managed the same way as every other host in the lab.

Bootstrap only covers hosts you create

Hypervisors and rented VPS instances never go through Stage 1 — they exist before the automation does. Those hosts need the ansible account, key, and sudo set up by hand before they can join the inventory. See Patch Automation → Step 2.

Stage 2 — Ongoing Maintenance (Every Week / Every Day)

Three playbooks run continuously in the background once a host is onboarded, on independent, offset schedules:

Playbook	Targets	Frequency	Purpose
Config Backup	All hosts	Daily (1 AM)	Archives critical app/system config to the control node
Patch Automation	Application containers	Weekly (Sun 3 AM)	Patches packages, prunes Docker images, reboots when required
Infra Patch	Hypervisors, public VPS	Weekly (Sun 4 AM)	Patches packages, reports reboot need instead of rebooting

They're deliberately scheduled a few hours apart within the same maintenance window — backup runs before patching, so if a patch cycle ever breaks something, there's always a same-day config snapshot to fall back to rather than relying on whatever the last weekly backup happened to catch.

# Daily config backup — 1:00 AM
0 1 * * * ansible-playbook -i hosts.ini backup_configs.yml

# Weekly prune — Sunday 2:00 AM
0 2 * * 0 ansible-playbook -i hosts.ini prune_backups.yml

# Weekly patch, application containers — Sunday 3:00 AM
0 3 * * 0 ansible-playbook -i hosts.ini update_lab.yml

# Weekly patch, infrastructure — Sunday 4:00 AM
0 4 * * 0 ansible-playbook -i hosts.ini update_infra.yml

All playbooks target the same inventory (hosts.ini) — a host is defined once, and which playbook picks it up depends only on which group it's in.

Not Every Host Should Be Patched the Same Way

The instinct is one group, one playbook, everything patched identically. That breaks the first time a hypervisor reboots itself at 3 AM and takes every container with it.

The inventory is split by how much a bad run costs, and each tier gets a playbook matched to that risk:

Tier	Group	Blast radius of a bad run	Reboot policy
Application	`linux_servers`	One service down	Automatic, when required
Hypervisor	`proxmox`	Every guest on that host down	Manual only, reported not performed
Public VPS	`vps`	Externally visible outage	Manual only, reported not performed

Two rules fall out of this split, and both matter:

Every group needs its own [groupname:vars] block. Group variables are not inherited. A new group without one falls back to connecting as root, which fails on any host with PermitRootLogin no and looks exactly like a broken SSH key.
A host in no group is a host in no playbook. Moving proxmox and vps out of the main group means nothing patches them until you deliberately write a play that does. That silence is intentional, but it's also easy to forget about.

Stage 3 — Recovery (When Something Breaks)

This is the payoff for maintaining Stage 2 consistently. When a service breaks — a bad config edit, a corrupted upgrade, a container that needs rebuilding — the recovery path is:

If it's a config problem → pull the most recent archive from /root/ansible-backups/<host>/, extract, and restore the specific file(s) needed (see Config Backup → Restoring).
If it's a whole-container problem → rebuild the LXC from a Proxmox template, run the Bootstrap Playbook against it, then restore configs from the last backup before reintroducing it to service.

Either path returns the host to Stage 2 — back under normal patch and backup coverage — rather than requiring anything to be reconfigured by hand.

Rebuilding a host at the same IP breaks the control node's SSH trust

A new machine at an existing IP presents a different host key, and SSH refuses to connect. Clear the stale fingerprint and reinstall the automation key before expecting Stage 2 to resume. See Handling a Changed Host Key.

Why This Order Matters

Each stage depends on the one before it actually happening:

Patch and backup playbooks assume the ansible account and SSH key already exist — skip bootstrap, and neither playbook can even connect.
Recovery assumes backups have been running long enough to have something recent to restore — skip consistent Stage 2 runs, and Stage 3 has nothing to work with.

The lesson from building this out: automation is only as good as its weakest, most-skipped stage. A host that got manually configured and never went through bootstrap is a host that's quietly missing from the whole system — it won't show up as a failure anywhere, it'll just silently not be covered.

The same failure mode applies to hosts that are in the inventory but fail every run. A host out of disk, or one whose home directory permissions broke, fails with an UNREACHABLE line that scrolls past in a log nobody reads. The rest of the run succeeds, so nothing looks wrong. Make a failure count part of the routine:

grep -c "UNREACHABLE\|fatal:" /var/log/ansible-patching.log

One Inventory, Two Purposes

Worth calling out explicitly: bootstrap_hosts.ini and hosts.ini are not the same file, and that's intentional:

bootstrap_hosts.ini — short-lived, root/password auth, used once per host and then that entry is removed
hosts.ini — the permanent inventory, key-based auth, used by every recurring playbook

A host should only ever exist in one of these at a time. If a host is still listed in bootstrap_hosts.ini after it's been through Stage 1, that's a sign the promotion step (moving it to the main inventory) got missed.

Name hosts in the inventory, don't list bare IPs

An inventory of bare IP addresses contains no record of what each address actually is. When an IP gets reassigned to a different application, the playbook patches the new machine under the old entry and nothing in the output tells you.

Named entries (authentik ansible_host=192.168.1.20) make the recap readable, make an IP change a one-value edit, and make a mismatch visible. See Patch Automation → Step 5 for a script that generates a named inventory from an existing IP-only one.

At a Glance

Stage	Playbook	Targets	Trigger	Auth Method
0 — Bootstrap	`bootstrap_lxc.yml`	New LXCs	Manual, once per host	Root + password
2 — Backup	`backup_configs.yml`	All groups	Cron, daily 1 AM	`ansible` user + SSH key
2 — Patch	`update_lab.yml`	`linux_servers`	Cron, weekly Sun 3 AM	`ansible` user + SSH key
2 — Infra Patch	`update_infra.yml`	`proxmox`, `vps`	Cron, weekly Sun 4 AM	`ansible` user + SSH key
3 — Recovery	Manual restore + re-run Stage 0/2	As needed	As needed	`ansible` user + SSH key

What's Next

This covers the lifecycle for general-purpose Linux hosts. Natural extensions from here:

A health check playbook to catch problems between scheduled patch/backup runs rather than only discovering them after the fact
Failure alerting — a task that posts the PLAY RECAP somewhere you'll actually see it, closing the silently-failing-host gap that log review alone doesn't
Role-specific bootstrap variants (e.g. a Docker-ready bootstrap for containerized services) once enough hosts share a common "type"
Moving backup_root_local off the control node entirely, so Stage 3 recovery doesn't depend on the control node itself surviving