A close that would not stick
I was about to clean up a third of my task board, and the code that filed those rows would have put them all back.
By Andrew Pyle
A dedup check is only as good as the list of states it is willing to look at.
01Before the cleanup
A third of the board was advice
I was about to collapse a pile of rows on my task board. About a third of the open board was advisory lines, filed by two commands that turn site audits into suggested tasks. Most of those came from one sweep six days earlier, when a new portfolio cadence ran the per site audit across every site I own and filed the top few items for each. Its commit noted that every site was processed with zero failures. That was true. It was also how the board filled up.
The plan was to fold those per site rows into a few rollups. That same cleanup has a story of its own, involving 132 duplicate tasks and a tasks page that crashed. This one is about the step before it. A cleanup only holds if the next filing run agrees with it, so I needed to know how the filers decided that a row already existed.
02Two status words
The check only saw open work
Both filers asked the same question before creating a row: is there already a task for this site with this exact title, in a status of open or in progress. If yes, skip it. If no, file it. The key was site plus title, checked against live statuses only.
Closing a row in my system means moving it to superseded or cancelled. Neither of those is open or in progress. So the moment I closed a row, it dropped out of the dedup check, and the next run would see no match and file a fresh copy. Every row I folded into a rollup would have come back on the next filing run, one per site, as if the cleanup had never happened.
Nothing about this throws an error. The filer reports what it filed, and the report is accurate. Closing was a state change the dedup had never been told about. The rows were advice in the first place, meant to propose rather than act, which is why a refiled copy is noise rather than work.
03Advice nobody can act on
Forty eight rows asked a human to wait for a machine
Reading the rows turned up a second problem. Forty eight of them said, in their own text, that another tracker will automate this check. The recommendation named a different system as the owner of the work. No person had anything to do, and the filer had put it in front of a person anyway.
In one of the filers this also cost real slots. It caps how many rows it files per site, and it applied the cap before throwing anything out. A line that could never be acted on could take the place of one that could.
04One shared rule
Count the closes, but not the done
Both filers now call one small shared helper, 27 lines, so the rule lives in one place instead of two copies. It has two functions. One decides whether a message is worth filing at all: empty text is not, and text containing will automate this check is not, matched without regard to case. The other decides whether a row was already filed, and it now looks at four statuses: open, in progress, superseded and cancelled. In the filer with the cap, the fileable check now runs before the cap is applied.
REFILE_BLOCKING_STATUSES = ('open', 'in_progress', 'superseded', 'cancelled')
_NON_ACTIONABLE_MARKERS = ('will automate this check',)Done is deliberately missing from that list. The module gives the reason in one comment.
A recommendation that still fires after 'done' was not done.
If I mark a row done and the next audit produces the same line, the work did not land, and I want the row back. It is the one decision in the change I flagged for review, because it is the one a reasonable person could argue the other way.
Five new tests cover the rule. The self automating line is skipped in both filers. A superseded row and a cancelled row are each left alone. A done row is refiled when the recommendation still fires. With the old status list and no marker check, four of the five fail. With the fix, the two filers' test files pass seventeen of seventeen.
05Not yet proven
The fix still has to survive a real run
I have not seen this work in production yet. The change left the existing rows alone, because collapsing them is the cleanup's job. The check I set for after it ships is simple: after the collapse, run one filer for one site with filing turned on, and it should report the lines as skipped because they already exist. The regular weekly run never files tasks, so only the opt in filing path changes.
The wider question is still open. These were two filers. Other code in the same repo files tasks too, and I did not audit how each of them dedupes. Four days later a different filer turned out to mint duplicate tasks in its own way, and needed its own fix. Before I trust a cleanup now, I ask two things of every filer that could undo it: which statuses it looks at, and which of my decisions it cannot see.
Related
writing
A queue with no backoff floods on every tick
My approval queue filled with the same pull request on every tick, and the thing it kept trying to merge could never merge.
writing
A dry run that spent money
The flag meant to make a rehearsal safe was making paid model calls and writing real rows.
writing
Dead is not asleep
A liveness alarm kept telling me to revive a laptop that was healthy and asleep, until I changed what the alarm counts as dead.
writing
Active was measuring the wrong thing
My overnight builder thought I was awake from 1 to 7 AM, because the signal it trusted for awake was being touched by something that was not me.
writing
The publishing pipeline that lied to me for four days
A green log, a stale site, and the difference between a job that ran and a job that did the thing.