feat(healthcheck): heartbeat before the security-gate stage (cap tick silence at ~30m, not ~55m) #360

New issue

Closed

opened 2026-07-06 19:37:02 +00:00 by igor · 0 comments

igor commented

2026-07-06 19:37:02 +00:00

Collaborator

Why

The dead-man's-switch heartbeat (hc_ping heartbeat, added in #348 for #297) fires only at the top of each tick. But a tier-1 tick runs two bounded model calls back-to-back — the build (TICK_TIMEOUT = 30m) then the security-gate review (claude_call … 1500 = 25m) — with no overall tick cap. So a perfectly healthy tick can go ~55 minutes between heartbeats, which forces the healthchecks.io grace window wide (~60m) and delays detection of a real stoppage. Goal: never more than one long stage (~30m) of silence.

What

Emit hc_ping heartbeat at the start of each long model-call stage that currently follows another long call in the same tick — concretely, immediately before each security_gate invocation (find them by the call signature, line numbers drift):

tier-1 issue flow: security_gate "$WORKTREE" "$PR_BASE" "security-gate-issue" (~bin/tick.sh:4300)
PR-review flow: security_gate "$PR_WORKTREE" "$PR_HEAD" "security-gate-pr-review" (~bin/tick.sh:3589)

That makes the max gap between heartbeats the longest single stage (~30m) instead of the sum (~55m). General rule: if any other stage can run a full-length model call after a prior long call in the same tick (e.g. a derive_pr_body fallback), ping before it too.

Must / must-not

hc_ping already no-ops silently when HEALTHCHECK_HEARTBEAT_URL is unset (lib/healthcheck.sh), so this is safe whether or not the check is configured. Keep it fire-and-forget — a ping must never affect the tick's exit or timing.
Do NOT add a background/periodic pinger — that's a separate, heavier option, explicitly out of scope here.
Use the existing hc_ping heartbeat call; don't invent a new ping type or URL.

Done when

A hc_ping heartbeat precedes each security_gate call in the tier-1 and PR-review paths.
make test passes. Nice-to-have: a lightweight source-assertion in bin/test-*.sh (in the spirit of check-sync's tick.sh greps) verifying a heartbeat precedes each long stage, so the guarantee can't silently regress.

## Why The dead-man's-switch heartbeat (`hc_ping heartbeat`, added in #348 for #297) fires only at the **top of each tick**. But a tier-1 tick runs **two** bounded model calls back-to-back — the build (`TICK_TIMEOUT` = 30m) then the security-gate review (`claude_call … 1500` = 25m) — with **no overall tick cap**. So a perfectly healthy tick can go **~55 minutes** between heartbeats, which forces the healthchecks.io grace window wide (~60m) and delays detection of a real stoppage. Goal: never more than **one long stage (~30m)** of silence. ## What Emit `hc_ping heartbeat` at the start of each long model-call stage that currently follows another long call in the same tick — concretely, **immediately before each `security_gate` invocation** (find them by the call signature, line numbers drift): - tier-1 issue flow: `security_gate "$WORKTREE" "$PR_BASE" "security-gate-issue"` (~`bin/tick.sh:4300`) - PR-review flow: `security_gate "$PR_WORKTREE" "$PR_HEAD" "security-gate-pr-review"` (~`bin/tick.sh:3589`) That makes the max gap between heartbeats the longest **single** stage (~30m) instead of the **sum** (~55m). General rule: if any other stage can run a full-length model call *after* a prior long call in the same tick (e.g. a `derive_pr_body` fallback), ping before it too. ## Must / must-not - `hc_ping` already **no-ops silently** when `HEALTHCHECK_HEARTBEAT_URL` is unset (`lib/healthcheck.sh`), so this is safe whether or not the check is configured. Keep it **fire-and-forget** — a ping must never affect the tick's exit or timing. - Do **NOT** add a background/periodic pinger — that's a separate, heavier option, explicitly out of scope here. - Use the existing `hc_ping heartbeat` call; don't invent a new ping type or URL. ## Done when - A `hc_ping heartbeat` precedes each `security_gate` call in the tier-1 and PR-review paths. - `make test` passes. Nice-to-have: a lightweight source-assertion in `bin/test-*.sh` (in the spirit of `check-sync`'s tick.sh greps) verifying a heartbeat precedes each long stage, so the guarantee can't silently regress.

igor self-assigned this

2026-07-06 20:17:35 +00:00

igor referenced this issue from a pull request that will close it,

2026-07-06 20:22:06 +00:00

chore: feat(healthcheck): ping the heartbeat before each security_gate call, capping tick silence at ~30m instead of ~55m #362

igor added spent time

2026-07-06 20:22:07 +00:00

4 minutes 23 seconds

igor removed their assignment

2026-07-06 20:22:07 +00:00