We launched a public status page on October 10. Two days earlier the history it now shows has five short red marks: four on getgordonai.com, one on zacmillersells.com, each exactly two minutes long. This is what those marks are, what they are not, and what we changed. Nothing in the history was edited.
What the page shows
The page is a second copy of Gatus, the same open-source monitor we run internally, fed only the public URLs. Before launch it had no history of its own, so we imported the internal monitor's results for the four sites it had already been checking every two minutes since October 3. That import is why October 8 is on the page at all. Each check that fails shows as two minutes of red, because two minutes is the gap to the next check.
| site | failed checks on Oct 8 (UTC) | what the check actually got |
|---|---|---|
| getgordonai.com | 13:04, 13:28, 14:52, 15:32 | HTTP 200 every time, in 5.12 to 5.21 s |
| zacmillersells.com | 15:32 | no response within the 10 s client timeout |
The check for the hub has two conditions: status 200, and response under 5 seconds. All four failures passed the first and missed the second by about 150 milliseconds.
What was actually slow
Not the sites. The monitor's own name lookups.
The box that runs the monitor resolves names through two Pi-holes on the home network, primary first. The primary is a Raspberry Pi Zero 2 W on Wi-Fi. On October 8 its radio had power saving on, and a Wi-Fi radio in power save drops packets while it sleeps. A dropped DNS query costs the resolver's 5 second timeout before the second Pi-hole is asked. Add a normal 100 to 200 ms page fetch and you get 5.1 to 5.2 seconds, which is the number in the table.
The internal monitor has a separate check that asks that Pi-hole to resolve a name once a minute. On October 8 that check failed 62 of 558 times, about 11 percent. The day before and the day after it looked like this:
| day (UTC) | resolver checks | failed |
|---|---|---|
| Oct 8 | 558 | 62 |
| Oct 9 | 1,440 | 0 |
| Oct 10, to 20:00 | 1,208 | 1 |
The fix was two commands on the Pi on the evening of October 8: turn power saving off on the wireless interface and tell NetworkManager to keep it off. Four lines of evidence point the same way: the hub's four failures were all 200s at just over the limit, the one client-site failure landed in the same minute as a hub failure, the resolver check was failing all day, and the day after the fix was clean.
What we cannot prove is that no visitor saw anything. A visitor's browser resolves our names through their own provider, not through a Pi in our house, so the mechanism above does not reach them. But the single zacmillersells.com timeout has no breakdown behind it, only a 10 second silence, and it stays on the page as a failure.
What changed besides the Pi
- Our main server now has both Pi-holes as resolvers, in that order, instead of only the primary. A dropped query there now costs a 5 second stall and a fallback instead of a failed lookup.
- The fleet health check for the primary Pi-hole only asked "does it answer at all" and stayed green through the whole day. It now also counts dropped single-try probes and warns when 6 of the last 30 fail.
- The first planned fault-injection exercise targets the second Pi-hole, and its preconditions now require the primary to be healthy first, because the main server leans on the second one as a fallback.
What we are not doing
We are not removing monitor-side failures from the uptime numbers. At launch the trailing 7-day figures on the page were 99.94 percent for getgordonai.com and 99.90 percent for zacmillersells.com, and October 8 is inside both. If the monitor cannot reach a site, the page says so, whatever the reason. The alternative is a page that only shows the failures we agree with.
What is next on this lane
A game day against the second Pi-hole, scheduled when there is a quiet window, to measure how long detection and recovery actually take rather than guessing. The baseline for that was taken on October 8 as well, from the alert history: for the one failure class with enough samples (72 DNS-related episodes on scheduled jobs), time to detect ran 5 minutes typically and 10 minutes at the 90th percentile, time to recover 10 minutes typically and 30 at the 90th, and every one of the 71 closed episodes healed on its own. Those are the numbers a drill has to beat.