A backup system that could not restore

Symptom

Backups ran nightly and every job was green. Then we tested a restore for the first time on a staging copy, which is a thing that should happen long before it is needed.

False lead

The first restore appeared to succeed. It had actually unpacked over the existing directory, so what came out was a merge of old and new files — a state that had never existed and that no test would recognise as wrong.

Cause

Three defects stacked: the restore merged instead of replacing, it deleted the current version before verifying the archive, and there was no way back if the archive turned out to be bad.

Fix

Rewritten as verify → snapshot the current state → unpack to a staging directory → atomic rename. The archive is checked with gzip and a full listing before anything on disk is touched, and the previous version is kept until the swap succeeds.

—A backup nobody has restored is not a backup. Schedule the restore, not just the backup, and report the result.