October 28, 2026·4 min read

The filter nobody wired up

A per-Mac health read told me every machine was fine. One of them had been failing for a month.

By Andrew Pyle

Ask for a machine that does not exist, and a working filter gives you nothing back.

01A nonexistent machine

I asked production for sessions from nowhere

mac_hostname=no_such_mac_xyz. I sent that filter to my own sessions API, asking for a machine I do not own and have never named. A working filter answers with an empty list. Mine answered with every session in the table, and the newest of them came from a different Mac.

GET sessions?mac_hostname=no_such_mac_xyz
expected: 0 rows
got:      every session, the newest row from a different Mac

That endpoint sat under a health check I trusted. The check walks each of my Macs and asserts that the hook relay and a few guard hooks are wired, present, and current against main, and that the relay produced a session row in production for that machine recently. It is read-only, one GET, and it exits 1 on any failure. For a long time it answered green for every Mac.

The relay line read the first row it got back and checked that row's last activity time. Since the filter did nothing, the first row belonged to whichever Mac had posted most recently. Any Mac posting anything made every Mac look fresh. The per-machine check had been one global question with a per-machine label on it.

# relay: fired, before the fix
# the request carried ?mac_hostname=<device>, and the API dropped it
last_seen = rows[0]["last_activity_at"]   # whichever Mac posted last

02What the green covered

One Mac had been failing for a month

The cause was a single missing piece of configuration. The viewset declared its filter fields, but no filter backend was configured, so Django REST framework never applied them. The queryset was never narrowed. Nothing raised an error. The parameter was accepted and then dropped.

While the check was green, one of my Macs had been failing its relay flushes since 2026-09-09. There were about 98,000 failures, each one rejected with HTTP 413, request body too large. It still read as fresh, because another machine's rows kept answering for it. It was not the only one. A second Mac's relay had been failing bulk flushes since the end of September, and an earlier version of the check stayed green through that too, because session rows still updated through a separate hook path.

The relay got its own fix: smaller chunks, oversized events trimmed with a marker, a single event that is still too large dropped with a warning, and a stop on the first retryable failure with the rest put back in the spool so nothing is lost. On one Mac's real backlog, 1,000 queued events drained to 21 in 14 flushes of about five seconds each, with zero errors. Before the drain, a new draining check was red with 1,351 errors in an hour. That check now fails if the backlog passes 200 or there are five or more flush errors in an hour.

This was also not the first time my relay went quiet behind a healthy-looking board. In July it posted to a local server, and when that server stopped, posts were dropped silently. The board got zero sessions for three days, and 213 stale ghost sessions hid the gap. That is why I already had a liveness alarm that cross-checks a second, independent usage feed.

03Make it fail loudly

A wrong-machine answer is now an error

The change has two halves. The viewset now applies the hostname and repo filters itself, which only adds WHERE clauses, and a request with no filters returns exactly what it did before. The health check now refuses a row from the wrong machine. If it asks about one Mac and gets another's row back, it fails with a message that names both and ends in device filter ignored. A future regression in the filter turns the check red instead of quietly passing.

API returned a '<other>' row for mac_hostname='<device>' (device filter ignored)

It came to 13 lines of source and about 90 lines of tests. There are six new tests, four on the API and two on the check's guard, including one where an unknown device gets no rows and one where no filter still lists every device. With both source changes reverted, four of the six fail. I also searched every caller I have for anything passing those two filters and found none, so nothing had been leaning on the broken behavior.

The proof I wrote down for after the deploy is the same question that found the bug: ask production for no_such_mac and expect zero rows. I have not yet checked a third Mac's relay directly. Before I trust a filter now, I ask it for something that cannot exist. A filter that throws when it cannot run gets caught on the first day, but one that accepts the input and ignores it hands its mistake to every check built on top of it.

Have something you need built or fixed?

I build production Django / Next.js platforms and human-supervised AI-agent systems. Solo, senior, and fast. Tell me what you are building.

Start a project