Maintenance: reap stale headless-browser (chrome/playwright) processes — kill anything alive > 60 min #388

New issue

Closed

opened 2026-07-15 03:01:42 +00:00 by igor · 0 comments

igor commented

2026-07-15 03:01:42 +00:00

Collaborator

CoS-owned reliability ticket (Igor). Leaked headless-browser processes have choked the host several times (screenshot/visual-verification captures that never exit) and we've never built a guard. Add one.

What

At the top of each maintenance tick, sweep for browser-automation processes and SIGKILL any that have been alive longer than the threshold — no questions asked.

Why this is safe to be aggressive

This is a headless server. There is no desktop browser — every chrome/chromium/headless_shell/playwright process is automation (screenshots, visual self-verification, deploy smoke). A legitimate capture finishes in seconds to low single-digit minutes. Anything alive for an hour is a leak, full stop.

Threshold (hardcoded, not a knob)

STALE_BROWSER_SECS = 3600 (60 minutes). Josh's bar was "1–2 hours is too long"; 60 min is the tight end and still ~10× any real run. Hardcode the constant per our strong-opinions convention — do not add an env var.

Heuristic (which processes to target)

Match on process name / cmdline. Kill when age (etimes) >= 3600 AND the command matches an automation signature:

binary basename in { headless_shell, chrome, chromium, chromium-browser, chrome_crashpad_handler }
or cmdline contains any of: --headless, --remote-debugging-port, --user-data-dir=/tmp, playwright, puppeteer, ms-playwright

Kill the whole process tree / process group (chrome forks a renderer/GPU/zygote tree — killing just the parent orphans the children). Prefer kill -9 -<pgid> or walk children by PPID.

Guardrails

Never touch the harness's own claude / node main process or its PPID chain.
Only kill procs at/over the threshold — younger automated runs are left alone.
Idempotent: safe to run every tick; a no-op when nothing is stale (the normal case).

Observability

When it kills something, log one line per victim: pid, etime, and a truncated cmdline (so it surfaces in logwatch and we can actually confirm the reaper is doing its job). Log nothing when it's a no-op.

Placement

Fold into the existing maintenance sweep (top of the tick) — least code, self-healing every cycle. Do not stand up a separate systemd timer unless the tick cadence turns out too slow to matter; default to folding in.

Tests (required)

Per our "tests for harness work" rule: extract the decision into a pure predicate — given a parsed ps table (pid, ppid, etimes, cmd), return the set of pids to kill — and unit-test it in bin/test-*.sh:

stale headless_shell (etimes 4000) -> killed
fresh headless_shell (etimes 30) -> spared
stale non-browser (e.g. a 2h node build) -> spared (signature must match)
stale chrome with --headless -> killed
the harness's own claude/node proc -> never selected

The test must exercise selection only (mock table), never actually kill.

Acceptance

Reaper runs each maintenance tick, kills stale browser trees, logs victims, no-ops silently otherwise.
Predicate is unit-tested with the cases above and CI (Validate) passes.

**CoS-owned reliability ticket (Igor).** Leaked headless-browser processes have choked the host several times (screenshot/visual-verification captures that never exit) and we've never built a guard. Add one. ## What At the top of each maintenance tick, sweep for browser-automation processes and **SIGKILL any that have been alive longer than the threshold — no questions asked.** ## Why this is safe to be aggressive This is a headless server. There is **no desktop browser** — every chrome/chromium/headless_shell/playwright process is automation (screenshots, visual self-verification, deploy smoke). A legitimate capture finishes in seconds to low single-digit minutes. Anything alive for an hour is a leak, full stop. ## Threshold (hardcoded, not a knob) `STALE_BROWSER_SECS = 3600` (60 minutes). Josh's bar was "1–2 hours is too long"; 60 min is the tight end and still ~10× any real run. Hardcode the constant per our strong-opinions convention — do **not** add an env var. ## Heuristic (which processes to target) Match on process name / cmdline. Kill when age (etimes) >= 3600 **AND** the command matches an automation signature: - binary basename in { `headless_shell`, `chrome`, `chromium`, `chromium-browser`, `chrome_crashpad_handler` } - **or** cmdline contains any of: `--headless`, `--remote-debugging-port`, `--user-data-dir=/tmp`, `playwright`, `puppeteer`, `ms-playwright` Kill the **whole process tree / process group** (chrome forks a renderer/GPU/zygote tree — killing just the parent orphans the children). Prefer `kill -9 -<pgid>` or walk children by PPID. ## Guardrails - Never touch the harness's own `claude` / `node` main process or its PPID chain. - Only kill procs at/over the threshold — younger automated runs are left alone. - Idempotent: safe to run every tick; a no-op when nothing is stale (the normal case). ## Observability When it kills something, log one line per victim: `pid`, `etime`, and a truncated cmdline (so it surfaces in logwatch and we can actually confirm the reaper is doing its job). Log nothing when it's a no-op. ## Placement Fold into the existing maintenance sweep (top of the tick) — least code, self-healing every cycle. Do **not** stand up a separate systemd timer unless the tick cadence turns out too slow to matter; default to folding in. ## Tests (required) Per our "tests for harness work" rule: extract the decision into a pure predicate — given a parsed ps table (pid, ppid, etimes, cmd), return the set of pids to kill — and unit-test it in `bin/test-*.sh`: - stale headless_shell (etimes 4000) -> killed - fresh headless_shell (etimes 30) -> spared - stale non-browser (e.g. a 2h `node` build) -> spared (signature must match) - stale chrome with `--headless` -> killed - the harness's own claude/node proc -> never selected The test must exercise selection only (mock table), never actually kill. ## Acceptance - Reaper runs each maintenance tick, kills stale browser trees, logs victims, no-ops silently otherwise. - Predicate is unit-tested with the cases above and CI (`Validate`) passes.

igor added the

Kind/Enhancement

Priority

High

labels

2026-07-15 03:01:42 +00:00

igor self-assigned this

2026-07-15 03:03:22 +00:00

igor referenced this issue from a pull request that will close it,

2026-07-15 03:12:07 +00:00

feat: reap stale headless-browser process trees at the top of the maintenance tick #389

igor added spent time