Lower cascade stages starve — 0 CEO/seo/maintenance runs in 2.5h #441

New issue

Closed

opened 2026-07-28 03:01:09 +00:00 by igor · 0 comments

igor commented

2026-07-28 03:01:09 +00:00

Collaborator

The cascade starves its own lower stages. Measured 2026-07-27: zero cascade-stage lines (ceo:, seo:, sports:, maintenance:) in a 2.5-hour window — not one tick got deep enough to reach them, because automerge, deploy-watching, PR review and issue work consumed every tick higher up.

That is not academic. igor#435 shipped a same-tick path for acting on board steering left on a CEO digest, and its responsiveness depends entirely on a tick reaching do_ceo_tick. A fast path behind a stage that goes unreached for hours is not fast.

Related measurement from the same window: 74 ticks, 58% utilization, median tick 79s, and only 9 of 74 claimed an issue. An idle tick still costs ~70s because every tick re-runs the whole fleet preamble before doing anything.

Decision: fairness, not reordering. Do not simply move the CEO stage earlier — that just starves whatever ends up last. A stage that has been skipped for N consecutive ticks should get priority on the next one. The cascade keeps its normal order when nothing is starving.

Explicitly NOT in scope: splitting the loop into separate services. That was designed and declined — both loops would write discretionary-state.json (36 call sites) through a non-atomic jq+mv, and the hazard is not worth it.

Also worth doing here since it is one line each and the measurement above is the justification:

AccuracySec=1s on agent.timer. It currently uses the systemd default of 1min, which batches wakeups and adds up to 60s of latency per fire on a machine that is not battery powered.
OnUnitInactiveSec from 1min to ~15s.

Acceptance criteria

A stage skipped for N consecutive ticks runs on the next tick regardless of what is above it.
A log line names when a stage is being prioritised because it was starved, so this is observable rather than inferred.
agent.timer sets AccuracySec=1s and a shorter inactive gap.
Tests cover the starvation counter and the priority decision.

The cascade starves its own lower stages. Measured 2026-07-27: **zero** cascade-stage lines (`ceo:`, `seo:`, `sports:`, `maintenance:`) in a 2.5-hour window — not one tick got deep enough to reach them, because automerge, deploy-watching, PR review and issue work consumed every tick higher up. That is not academic. igor#435 shipped a same-tick path for acting on board steering left on a CEO digest, and its responsiveness depends entirely on a tick reaching `do_ceo_tick`. A fast path behind a stage that goes unreached for hours is not fast. Related measurement from the same window: 74 ticks, **58% utilization**, median tick 79s, and only 9 of 74 claimed an issue. An idle tick still costs ~70s because every tick re-runs the whole fleet preamble before doing anything. **Decision: fairness, not reordering.** Do not simply move the CEO stage earlier — that just starves whatever ends up last. A stage that has been skipped for N consecutive ticks should get priority on the next one. The cascade keeps its normal order when nothing is starving. Explicitly NOT in scope: splitting the loop into separate services. That was designed and declined — both loops would write `discretionary-state.json` (36 call sites) through a non-atomic jq+mv, and the hazard is not worth it. Also worth doing here since it is one line each and the measurement above is the justification: - `AccuracySec=1s` on `agent.timer`. It currently uses the systemd default of **1min**, which batches wakeups and adds up to 60s of latency per fire on a machine that is not battery powered. - `OnUnitInactiveSec` from `1min` to ~15s. ## Acceptance criteria - A stage skipped for N consecutive ticks runs on the next tick regardless of what is above it. - A log line names when a stage is being prioritised because it was starved, so this is observable rather than inferred. - `agent.timer` sets `AccuracySec=1s` and a shorter inactive gap. - Tests cover the starvation counter and the priority decision.

igor referenced this issue

2026-07-28 16:03:58 +00:00

fix: stop paying systemd 1-minute timer accuracy slop #447

igor referenced this issue

2026-07-28 16:05:20 +00:00

fix: stop paying systemd 1-minute timer accuracy slop #447

igor referenced this issue from a commit

2026-07-28 16:30:51 +00:00

fix: let a starved cascade stage jump the queue

igor referenced this issue from a pull request that will close it,

2026-07-28 16:31:10 +00:00

fix: let a starved cascade stage jump the queue #448