Running a Restore Drill on a Proxmox Homelab

My backup jobs have been green for a long time. That tells me the jobs ran. It does not tell me I can get a service back. I've been calling the strategy "not fully tested," which is a polite way of saying I don't know yet.

So I'm building a restore drill: a short, repeatable procedure that proves a backup is usable, and writes down how long it took. This post covers the structure I'm settling on. I haven't run it against every service yet, so treat it as a working plan, not a finished answer.

What "restored" has to mean

The first problem is the definition. A restore that completes without errors is not the same as a service that works. I'm using three levels:

1. The archive is readable. The backup can be opened and its checksums verify.
2. The data is present. A VM boots, or a dataset has the files I expect.
3. The service works. The app answers, the database is consistent, and a real user action succeeds.

Most homelab "restore tests" stop at level 1 or 2. Level 3 is the one that matters when the Wi-Fi is down and someone is asking why. It's also the slowest to check, so I only do it for the services the household notices.

Rules for the drill

A few constraints keep the drill from becoming its own outage:

• Never restore over the original. Restore to a new VMID or a scratch path.
• Isolate the network. A restored VM has the same hostname, and often the same static IP, as the live one. Put it on a bridge with no uplink.
• Disable autostart. A restored VM with onboot set will come back after the next host reboot, and I don't want that.
• Time it. The number I want is wall-clock time from "start restore" to "service verified." I don't have that number for anything yet.
• Clean up. Destroy the test VM at the end, or the drill leaves clutter behind.

Drill 1: a Proxmox VM from a vzdump archive

This is the shape of the script I'm using. Paths and IDs are placeholders. Adjust the archive location for your own backup storage.

#!/usr/bin/env bash
set -euo pipefail

SRC=105          # live VM being tested
TEST=9105        # scratch VMID
STORAGE=local-zfs
BRIDGE=vmbr99    # isolated bridge, no uplink
ARCHIVE=$(ls -t /mnt/pve/backup/dump/vzdump-qemu-${SRC}-*.vma.zst | head -n1)

start=$(date +%s)

qmrestore "$ARCHIVE" "$TEST" --storage "$STORAGE" --unique
qm set "$TEST" --onboot 0 --net0 virtio,bridge="$BRIDGE"
qm start "$TEST"

# requires qemu-guest-agent inside the guest
until qm agent "$TEST" ping 2>/dev/null; do sleep 5; done

echo "Booted in $(( $(date +%s) - start ))s from $ARCHIVE"

A few notes on why it looks like this:

• --unique generates new unique properties such as MAC addresses, so the copy doesn't collide with the original.
• qm set --net0 replaces the NIC definition and moves it to the isolated bridge. I'd rather lose the original NIC settings on a throwaway VM than risk a duplicate IP on the real network.
• The guest agent ping is a cheap "did it actually boot" check. It only works if the agent was installed before the backup, which is itself a useful thing to learn.

After that comes the level 3 check. For a service VM, that means logging in through the console, or from a scratch VM on the same isolated bridge, and exercising the app. Then clean up:

qm stop 9105
qm destroy 9105 --purge

Drill 2: ZFS snapshots and file-level recovery

For the NAS, the common failure is not "the pool is gone." It's "I deleted something three weeks ago." That is a snapshot question, and it's easy to test without touching the live dataset.

zfs list -t snapshot -r tank/photos | tail -n 5

# browse a snapshot read-only, no rollback needed
ls /tank/photos/.zfs/snapshot/<snapshot-name>/

# or clone it to a scratch dataset
zfs clone tank/photos@<snapshot-name> tank/restore-test

The drill is to pick a file at random, find it in an old snapshot, and copy it out. Then destroy the clone with zfs destroy tank/restore-test.

The harder test is the off-box copy. If snapshots are replicated with zfs send and zfs receive to another machine, the real question is whether I can restore from that copy, not from the local snapshots that die with the pool. A local snapshot is not a backup in the 3-2-1 sense. I'm treating the replicated copy as the one that counts.

Drill 3: file-level backups with restic

For application data that lives outside VM images, I'd use restic's own verification, then a real restore into a scratch directory:

restic check --read-data-subset=5%
restic restore latest --target /tmp/restore-test --include /srv/vaultwarden

restic check alone verifies repository structure. The --read-data-subset option also reads a portion of the actual data, which is slower but catches more. A different random subset each run means the whole repository gets covered over time. That's my understanding of how it behaves, so check the docs for your version.

Then open the restored data and confirm it's usable. For a password manager, I'd want to confirm the database opens, not just that the files exist.

Keep a drill log

The part I expect to skip unless I force myself: write down the result. A plain text file in Git works. Per drill I want:

• Date and what was restored
• Which backup (timestamp) was used
• Time to restore, and time to verified
• What broke or surprised me
• What I changed afterward

The "what broke" line is the point of the whole exercise. Every failed drill is a bug found while it was cheap.

What I haven't solved

• Databases. A VM-level backup of a running database may be crash-consistent, not application-consistent. I'm unsure which of my services need a proper dump taken before the backup runs.
• Encryption keys. If the backup is encrypted, the key has to exist somewhere that survives losing the cluster. I'm still working out where that lives.
• Cadence. Monthly for critical services feels right, but I'm not sure that's realistic for me. Quarterly might be the honest number.
• Bare-metal recovery. Restoring one VM is the easy case. Rebuilding the cluster from nothing is the one I'm avoiding thinking about, and I'd only trust my recovery plan once I've tried it.

Questions for you

How often do you run restore drills, and did you automate any of it? Do you verify at the service level, or stop once the VM boots? And has a drill ever turned up a failure that a green backup job was hiding? I'd like to hear those before I decide on a cadence.