What actually broke,
and how we found it.

Symptom

Outbound SMS stopped arriving. The provider’s API was reachable by ping, its status page was green, and every other outbound request from the same machine worked normally. The TLS handshake simply hung.

False lead

Everything pointed at the provider blocking our IP. We were composing the support ticket when we checked plain HTTP to the same host — it answered instantly. A blocked address does not answer on port 80 and hang on 443.

Cause

A path-MTU black hole. The TCP connection opened on small packets, then the first large frame of the handshake was dropped somewhere upstream and the ICMP "fragmentation needed" reply never came back. Nothing timed out; it just stopped.

Fix

MSS clamped to 1360 on the host, applied as a service so it survives a reboot. Confirmed by bracketing the failure between 1400 and 1420 bytes before and after.

—When a service is "down", find a second thing about it that still works. The difference between the two is the diagnosis.

Symptom

Several domains resolved perfectly from the office and failed on mobile networks. Refreshing helped sometimes. The nameservers were running, answering locally, and had not been touched.

False lead

A caching problem, obviously — so we lowered TTLs and waited. It changed nothing, which cost a day. The pattern was not cache: it was consistent per network, not per time.

Cause

Inbound DNS was being filtered upstream of the entire address range. Queries from some resolvers never reached the servers at all. Self-hosted authoritative DNS was not possible on that network, and no amount of configuration was going to fix it.

Fix

All zones moved to a managed anycast provider, with the records driven from our own control plane through their API so nothing is maintained by hand.

—Test from outside your own network, always. A resolver on your desk is the one client guaranteed not to reproduce the bug.

Symptom

Backups ran nightly and every job was green. Then we tested a restore for the first time on a staging copy, which is a thing that should happen long before it is needed.

False lead

The first restore appeared to succeed. It had actually unpacked over the existing directory, so what came out was a merge of old and new files — a state that had never existed and that no test would recognise as wrong.

Cause

Three defects stacked: the restore merged instead of replacing, it deleted the current version before verifying the archive, and there was no way back if the archive turned out to be bad.

Fix

Rewritten as verify → snapshot the current state → unpack to a staging directory → atomic rename. The archive is checked with gzip and a full listing before anything on disk is touched, and the previous version is kept until the swap succeeds.

—A backup nobody has restored is not a backup. Schedule the restore, not just the backup, and report the result.

Symptom

An internal query console had a strict allow-list: reads only, one table at a time, no file functions. The unit tests covering it were thorough and all green.

False lead

Because the tests passed, the feature looked finished. Test coverage is a statement about the code that was written, not about the code that was reached.

Cause

A second execution path for privileged users skipped the guard entirely. Sending two statements separated by a semicolon ran both. It was found by trying it against the live tool, not by reading the code.

Fix

The single-statement and read-only checks now sit in front of every path, for every role, with the privileged path holding no exemption at all. The attempt that found it became a test.

—Attack your own tool before shipping it. A green suite proves the paths you thought of are safe.

Symptom

A fix went out, verified in a private window, confirmed live. The client still saw the old behaviour, and so did anyone who had opened the site that week.

False lead

We spent a round hunting a build problem, because the served file on the server was demonstrably correct. It was — nobody was asking for it.

Cause

Stylesheets and scripts carried no version in their filenames and were being served with a seven-day cache. Returning browsers had no reason to ask again, so every deploy landed for new visitors only.

Fix

Code assets now revalidate on every request; only genuinely immutable files — fonts, images with hashed names — keep a long cache. Verified from a cold and a warm browser after each release.

—Verify in the state your users are actually in. A private window is the one visitor who has never been to your site.