For a long time my patching routine was: SSH into a VM, run apt upgrade, glance at the output, move on. It worked until it didn't, and the one time it didn't was a kernel update on a VM that happened to run DNS. Nobody in the house cares about my kernel version. They care that the Wi-Fi "stopped working."
This post covers the playbook I'm building to replace that routine. The goals are modest. Take a snapshot before touching anything. Patch one host at a time. Reboot only when needed. Check that the host came back before moving to the next one. I haven't run this for long, so treat it as a working draft. I'll flag the parts I'm unsure about.
If you want to see the manual version once before automating it, the Install Debian lab ends with refreshing the package list and installing the waiting updates.
You'll practice applying updates by hand on a fresh server, which is the exact step this playbook repeats.
Why not just unattended-upgrades
Unattended upgrades are a fine choice, and I still use them for security updates on a couple of VMs. My hesitation is about ordering and visibility. They patch whenever the timer fires, they don't know which VM is my resolver, and a failed run is easy to miss. A playbook runs when I run it, in the order I choose, and tells me what happened.
The tradeoff is obvious: a playbook only patches when I remember to run it. That's the same weakness I'm trying to escape. My current lean is both. Unattended security updates as the floor, the playbook for full upgrades on a schedule I control. I'm not certain that's the right split.
The inventory
The inventory carries one extra variable per host, the Proxmox VM ID, so the playbook knows which snapshot to take.
all:
children:
lab_vms:
hosts:
dns1:
vmid: 110
docker1:
vmid: 120
git1:
vmid: 130
devvm:
vmid: 140
Host order matters here. With serial: 1, Ansible works through hosts in inventory order, so I put the least critical VMs first. If something breaks, it breaks on the dev VM and not on the one the family depends on.
The playbook
Here is a trimmed version.
- name: Patch lab VMs
hosts: lab_vms
serial: 1
become: true
tasks:
- name: Snapshot the VM before patching
ansible.builtin.command: >
qm snapshot {{ vmid }}
pre_patch_{{ ansible_date_time.date | replace('-', '_') }}
delegate_to: pve1
changed_when: true
- name: Upgrade packages
ansible.builtin.apt:
update_cache: true
cache_valid_time: 3600
upgrade: dist
- name: Check whether a reboot is pending
ansible.builtin.stat:
path: /var/run/reboot-required
register: reboot_flag
- name: Reboot if needed
ansible.builtin.reboot:
reboot_timeout: 600
when: reboot_flag.stat.exists
- name: Confirm the host is healthy
ansible.builtin.command: systemctl is-system-running --wait
register: sysstate
changed_when: false
failed_when: sysstate.stdout not in ['running']
A few notes on why each piece is there.
Snapshot first, delegated to the node
The snapshot task runs on the Proxmox node, not the VM, which is what delegate_to: pve1 does. I used the plain qm snapshot command so the step is easy to read and to run by hand. There is a Proxmox snapshot module in the community.general collection, but I haven't used it, so I won't vouch for it.
The unsure part: the SSH user for the node needs permission to run qm, and I haven't settled whether a dedicated automation user is better than root with a key. Also, become: true applies to the delegated task too, so check that it does what you expect on your node.
A snapshot is a rollback point, not a backup. It lives on the same storage as the VM. It also grows as the disk changes, so don't leave old ones around. I delete them by hand after a few days of nothing breaking. Automating that cleanup is on my list.
The reboot check
Ubuntu creates /var/run/reboot-required after certain updates. Plain Debian doesn't do it by default, as far as I know. On Debian hosts the stat check will quietly say "no reboot needed" forever, even after a kernel update. I'd rather admit that than pretend the check is universal. For Debian VMs I'm considering rebooting unconditionally, or comparing the running kernel against the newest installed one. Haven't built either yet.
The health check
systemctl is-system-running --wait waits for boot to finish and reports running if every unit is fine. It reports degraded if any unit failed. That's a blunt check, since one broken timer unit will fail the whole thing. For me that's acceptable: a failed unit after patching is something I want to look at anyway. If it's too noisy, swap it for a check that targets the service, such as hitting a health URL with the uri module.
What stops the run
My reading of the docs is that when a host fails in a serial batch, Ansible aborts the remaining batches by default, because the failure rate exceeds the default max_fail_percentage of zero. That is the behavior I want. If docker1 doesn't come back, the playbook should not carry on to git1. I tested this once by deliberately failing a task on the first host, and the later hosts were skipped. One test isn't proof, so I'll keep watching it.
Things I'm deliberately not automating yet
- Container image updates. Compose stacks have their own update path, and I pin image tags. Mixing that into the OS playbook would blur the blast radius.
- The Proxmox nodes themselves. Patching the hypervisor takes down every VM on it. I do that manually, with a plan for where the VMs go.
- Timing. I run it on a weeknight when nobody is mid-movie. Running it from cron would remove the "someone is watching" check, and I'm not ready to give that up.
What I'd do differently
If I started again, I'd add a --check dry run to my routine before the real one. It's imperfect with command tasks, which don't run in check mode unless you tell them how to, but it still shows which hosts have pending upgrades.
The bigger lesson is that the snapshot is the part that changed my behavior. Knowing I can roll back in a minute makes me far less nervous about running the upgrade at all.
A few questions for those of you who patch more VMs than I do. Do you run full upgrades on a schedule, or only security updates? Have you ever rolled back from a snapshot after a patch run, and what broke? And how do you handle the DNS VM specifically: patch it first, last, or by hand?
Have a question about this guide?
Create a free account to ask the community and join the discussion. Members who run the real thing answer every day.