A deploy should drain, not kill

At 1:30 one afternoon a worker picked up my daily brief. At 1:33 a deploy restarted that worker. The brief had been running for about two and a half minutes, and it died with a worker-lost error naming signal 15. The next line in the log read: Suppressing deploy-noise alert.

There was no run record for it and no brief for the day. The brief runs once a day, so nothing tried again. Six automated agents read from it, and every one of them showed not firing. From the outside it looked like six things had broken at once. One thing had been interrupted, and the interruption landed on the single daily run with no margin to absorb it.

One alarm did go off, at 4:11, and it was right to. A stall check watches for a missing brief, and the brief was missing. So the system could tell me the brief never arrived. It had no way to tell me something had killed it.

My deploys ship every green build on main, so a restart landing on running work happens several times a day. The workers run under systemd, fourteen units of them, and none of those units said anything about how to stop. The default signals the whole control group, with a ninety second timeout. On a restart, every worker child gets SIGTERM at the same instant, and in-flight work does not get to finish.

Two things kept me from seeing it. My own failure handler deliberately filed this error as deploy noise and silenced the alert. And late acknowledgement was off, so a task killed mid-run was never redelivered.

When I counted, the worker journals held 125 of these kills in seven days. Most were short jobs that run often and ran again on their next cycle. Only eight that week were long jobs that run past 660 seconds, the drain window I would later set. The daily brief was the one job with no next cycle.

The first change was small and aimed at the one job. Instead of running once at 1:30, the brief now runs at half past each hour from 1 to 5 Central. A guard at the top returns early, reporting already briefed today, if a brief exists since Central midnight. A brief killed at 1:33 tries again at 2:30, then 3:30, 4:30, and 5:30, and the guard turns the later runs into no-ops once one lands.

The guard has to run before the rest of the job's work, or a catch-up run would repeat that work. There is still a gap: a run could die after the work and before the brief is written, and the retry would redo it. Unique constraints on what the job writes, one per agent per day and one per work key, stop that from producing duplicates. The cost is four extra runs a day, each one a single indexed existence query.

That shipped first, about twenty-seven minutes ahead of the real fix. It protected the brief and did nothing for the other kills.

KillMode=mixed
TimeoutStopSec=660
doppler run --forward-signals

Each deploy reinstalls the unit files, so one dropped line would quietly undo all of this. A regression test asserts that every worker unit keeps the three settings and that the deploy restarts workers in one call with at least 25 minutes to do it. One detail is slightly circular: the change's own deploy was the first to drain, since the new unit files are copied and reloaded before the workers restart. When it merged, I had not yet watched a drained restart in production. The check I wrote down is to count signal 15 lines in the worker journal after a deploy, and I have not recorded that count here.

journalctl -u '<worker units>' --since <deploy> | grep -c "signal 15"

The worse half was the monitor. I keep a log of every scheduled run and how it ended. For the same seven days that held 125 kills, that log held zero killed runs. A clean record from a log that cannot hold the bad news tells me nothing at all.

The reason was where the error fires. A worker-lost error raises the failure signal in the worker's main process, and there the request carries no name for the scheduled job. An ordinary exception raises it in the child, which has that context. My handler needed the name to write the record, so for a kill it wrote nothing.

Now a handler on task received remembers each scheduled task's name by task id in the main process, capped at 5,000 entries with the oldest evicted first. The failure handler falls back to that name when the request is empty. The alert stays silenced. The log records the kill, and my agents page counts deploy kills separately, outside the success rate. An out-of-memory kill arrives as signal 9, so it still counts as a real failure and still alerts.

Some of this is still open. Jobs that run longer than 660 seconds are still killed by a deploy, including a few that run for one to two hours. I chose not to turn on late acknowledgement for them. On Redis an unacknowledged task comes back after the visibility timeout, an hour by default, so a healthy task that ran past an hour would run twice, and a paid one would spend its model budget twice. That trades a rare loss for a new risk. While digging I also found that my schedule seeding ignored per-job time limits, so jobs that declared an hour or two were running at 600 seconds.

The habit I am keeping: a restart should let running work finish, and a monitor that has never once reported the thing it watches for may be unable to see it.