Sites that worked on a laptop and died on a phone

Symptom

Several domains resolved perfectly from the office and failed on mobile networks. Refreshing helped sometimes. The nameservers were running, answering locally, and had not been touched.

False lead

A caching problem, obviously — so we lowered TTLs and waited. It changed nothing, which cost a day. The pattern was not cache: it was consistent per network, not per time.

Cause

Inbound DNS was being filtered upstream of the entire address range. Queries from some resolvers never reached the servers at all. Self-hosted authoritative DNS was not possible on that network, and no amount of configuration was going to fix it.

Fix

All zones moved to a managed anycast provider, with the records driven from our own control plane through their API so nothing is maintained by hand.

—Test from outside your own network, always. A resolver on your desk is the one client guaranteed not to reproduce the bug.