fix: stop leaking http.server daemons from ticks (reaping only orphans) #428
No reviewers
Labels
No labels
Agent
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No assignees
2 participants
Notifications
Total time spent: 14 minutes 42 seconds
Due date
igor
14 minutes 42 seconds
No due date set.
Dependencies
No dependencies set
Reference
joshtronic/igor!428
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/418-reap-only-orphans"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What this PR does
Supersedes #422 -- carries all of its commits plus one review fix. Opened as a separate branch rather than pushing onto the bot's PR branch, which collides with the rework loop.
python3 -m http.serverdaemons from ticks (original #422 work)The review fix
http_reap_select_victimsread the ppid field and discarded it (while read -r pid _ etimes cmd), leaving age and cmdline as the only criteria. Any match older thanHTTP_REAP_STALE_SECSwas SIGKILLed regardless of parentage:An operator running
python3 -m http.serverto eyeball a build would lose it inside five minutes, to a sweep that now runs on every tick, with the only trace a line in the journal. That is a worse day than the leak being fixed.Now gated on
ppid == 1, using the field already being read. That is the leak's actual signature -- every orphan in igor#418 was reparented to init when its tick exited -- and it additionally spares a listener that a currently running tick legitimately owns, which the age check alone would have killed mid-verification.Test gap that hid it
Every row in
bin/test-http-reap.shusedppid 1, so the suite passed whether the guard existed or not. Added three rows differing ONLY in parentage, plus a live-parent sibling in the multi-row case, so this cannot regress silently.Test plan
bash bin/test-http-reap.shpasses -- 20 assertions, including the 3 new ppid rowsmake testpasses (24 suites)shellcheck lib/http-reap.sh bin/test-http-reap.shcleanpstable: selects nothing, correct -- the four orphans from igor#418 were manually reaped on 2026-07-24basename(it is ppid 1, so the new guard does not short-circuit that coverage)Not changed
browser_reap_sweepmoving from maintenance-only to every-tick stays as the original PR had it -- right direction, ~30x cadence increase on igor#388's predicate. Flagging rather than touching it.Closes #418
python3 -m http.serverdaemons from tickspython3 -m http.serverdaemons from ticks #422🤖 Review —
REQUEST_CHANGES(automated)CI for
3d54a9ee: successAdds a
python3 -m http.server-style reaper (lib/http-reap.sh+ unit tests + directive text) and hoists both it and the existing browser reaper out ofdo_maintenance_tickto run on every tick. The http predicate itself is careful work and the ppid fix over #422 is the right call — but the PR also carries an unrelated cadence change tobrowser_reap_sweepthat it explicitly declines to make safe, and one false-positive class in the new predicate.Blocking
bin/tick.sh:1222-1231/3047-3059— out-of-scope behavior change tobrowser_reap_sweep, at ~30x cadence, without the guard this PR was opened to add. Moving the browser sweep from maintenance-only to unconditional-every-tick is a change to igor#388's subsystem, not igor#418's; the description acknowledges it ("Flagging rather than touching it") but flagging doesn't put it in scope. Worse, the diff tolib/browser-reap.shis comments only, so as far as I can see from this diff its predicate is still age + cmdline with noppid == 1gate. That is precisely the bug this PR fixes for http servers — "the age check alone would have killed a listener a currently running tick legitimately owns" — now amplified 30x for browsers. Fix: either leavebrowser_reap_sweepinsidedo_maintenance_tickand hoist onlyhttp_reap_sweep(this PR's actual issue), or apply the same orphan guard tobrowser_reap_select_victimsin a separate PR. Either way the browser cadence change doesn't belong bundled here. I could not verify the browser predicate from the diff, which is itself a reason not to ship the cadence change in this PR.lib/http-reap.sh:96—ppid == 1is not the same as "orphaned" on a systemd host. Every systemd-managed service is ppid 1. A supervisedphp -S ... -t distunit, anhttp-serverunder a unit file, or any python service whose argv containshttp-serveris stale-by-definition (uptime ≫ 300s), ppid 1, and matches the signature — so this sweep SIGKILLs it every tick, and systemd restarts it, producing a flap loop whose only trace is a journal line. That's the exact failure mode the ppid guard was added to prevent, just moved to a different victim. At minimum this needs an additional discriminator (e.g. skip processes with a systemd cgroup //proc/$pid/cgroupnot under the agent's slice) or an explicit, documented statement that no supervised static server may exist on this host. A test row covering "ppid 1 but supervised" would pin whatever you choose.Needs an answer before merge
bin/test-http-reap.shis new but the Makefile isn't touched. The test plan claimsmake testpasses with 24 suites. If the runner enumerates suites explicitly rather than globbingbin/test-*.sh, this file is never executed by CI and the green status says nothing about it. Confirm the glob (or add the wiring).Non-blocking notes
http_reap_sweepand_http_reap_kill_treehave no coverage at all — understandable (they call realkill), but the row-lookup path (grep -E "^[[:space:]]*${pid}[[:space:]]"+read -r _ _ etimes cmd) is untested string handling in the logging path. Low risk since the regex is anchored, just noting the gap.setsid/nohup/a subshell that exits is ppid 1 while the owning tick is still alive, so it can still be killed mid-verification; conversely a leak whose parent shell survives is now never reaped. Both are defensible tradeoffs, but the header comment claims the guard "spares a listener a currently-RUNNING tick legitimately owns," which is only true when the tick is the direct parent. Worth softening the comment.lib/http-reap.sh:96, three new ppid rows T7A/T7B/T7C plus the2005live-parent sibling inMULTI, the-bashrow at T6). Assertion count reconciles at exactly 20. No fabrication found.AGENTS.mdandbin/lib/site-work-directive.mdare in scope and read well.Independent review by the harness on
claude-opus-5(effort: high). The human reviewer is requested once Igor has reviewed; a human still merges.🔧 Rework — round 1 (automated)
Addressed the review on
claude-opus-5at effort high — 2 new commit(s).🤖 Review —
COMMENT(automated)CI for
7f3f257a: successAdds a per-tick sweep (
lib/http-reap.sh) that SIGKILLs orphaned static-file servers, plus directive text telling the agent not to spawn them, wired intobin/tick.shwith a mock-table unit suite. The code itself reads carefully and the ppid/cgroup guards are well reasoned — but it's a destructive host-level kill running on every tick, and there are two things I can't verify from the diff, so I'm leaving this for a human rather than approving.What I checked and found clean
ppid == 1gate is genuinely present (lib/http-reap.sh,[ "$ppid" = "1" ] || continue) with three dedicated ppid rows (T7A/T7B/T7C) and a live-parent sibling (2005) in the multi-row case./system.slice/static-site.service/worker→static-site.service(spared),session-5.scope→ short-circuits to empty (reaped),.../app.slice/agent.service==HTTP_REAP_OWN_UNIT(reaped).while readloop — the/procread happens in a command substitution with its own redirection.set -esafety: everyA && B && continue/[ -z "$x" ] && return 0is mid-body, and thewhileloop's terminal status is 0 on all paths I could construct, sovictims=$(http_reap_select_victims ...)shouldn't abort a tick.grep -E "^[[:space:]]*${pid}..."is fed a pid already validated numeric; no untrusted interpolation, no secrets, no CI-config change.Things a human should weigh
bin/test-http-reap.shactually run bymake test? The diff adds the file but no Makefile / test-runner registration. If the runner globsbin/test-*.shthis is fine; if suites are listed explicitly, the entire new suite never runs in CI and the green check says nothing about it. This is the one thing that would change my verdict either way — please confirm.killed/spared/inline checks (T1, T2, T2B, T3, T3B, T3C, T3D, T3E, T4, T4B, T5A, T5B, the stderr check, T6, TB1, TB2, T7A–C, T8A–E, multi-row). The tests are real and more thorough than claimed — just note the number is off.nohup/setsid/disownedpython3 -m http.serveran operator left running is ppid 1 with no.servicecgroup, so it gets SIGKILLed 5 minutes later with only a journal line. The code documents this as deliberate ("under-reaping is the cheap failure" — but this is the over-reaping direction). Whether that's acceptable depends on whether this host is single-purpose; the human owns that call, not me.http_reap_sweep). Victims come from apssnapshot;_http_reap_kill_treekills$pidlater without re-confirming the cmdline still matches. The window is milliseconds, but the consequence is an unrelated SIGKILL. Cheap hardening: re-read/proc/$pid/cmdlineand bail if the signature no longer matches.browser_reap_sweepcadence was left as-is and explicitly flagged in the description and in thetick.shcomment. Good call; correctly out of scope.Follow-ups worth a ticket
HTTP_REAP_OWN_UNITis documented as a test seam but is env-readable in production; a stray export would silently narrow (or widen) the cgroup guard. Consider making the test override explicit instead (e.g. only honored when a test flag is set)._http_reap_cgroup_path/_http_reap_supervising_unitpair against a synthetic/proc-style fixture — the whole cgroup layer is stubbed out in the suite, so a parsing regression in0::handling would pass CI.Independent review by the harness on
claude-opus-5(effort: high). The human reviewer is requested once Igor has reviewed; a human still merges.