joshtronic/igor

Watch

Set up healthchecks.io heartbeat monitoring (overnight visibility): igor-ticking + task-finished pings #297

New issue

Closed

opened 2026-06-30 15:12:02 +00:00 by igor · 2 comments

igor commented

2026-06-30 15:12:02 +00:00

Collaborator

Why

Last night the tick timer sat stopped for ~8.5 hours and nothing surfaced it — we only caught it because I went looking this morning. We need a dead-man's switch: if igor stops ticking (or a tick hangs mid-task), you get paged, instead of silence. monit can't cover this — agent.service is a per-minute oneshot, not a long-running process monit can watch; a periodic-ping heartbeat is the right tool for a cron job.

Set up self-hosted healthchecks on the monitoring server and wire two pings out of the tick.

The two checks (your framing → healthchecks mechanics)

A. "igor is ticking" (heartbeat / dead-man's switch) — ping at the start of every tick. Since ticks fire every minute, this doubles as your "igor starting" signal and the liveness alarm: set period ~1 min, grace a few min, and if healthchecks hears nothing it alerts you. This is the piece that would have caught last night.

B. "finishing a task" — ping when a tick completes real work (a genuine # OUTCOME, not a no-op idle tick). Best as a start→success pair around the work: send /start when a do_*_tick begins working and the success ping when it returns. Then a tick that starts a task but never finishes (a crash/hang — the exact #291 pattern we just fixed) shows up as "started, never completed," and you get per-task duration too.

Work split

You (this ticket — the server + config):

Stand up healthchecks on the monitoring box (Docker compose + Postgres + a notification channel — email/ntfy/push to your phone).
Create the two checks; grab their ping URLs; pick period + grace for each.
Drop the ping URL(s) into the host .env at ~/.local/share/agent/.env.

Harness code (a follow-up bot PR once the URLs exist — I can take this):

lib/healthcheck.sh helper, e.g. hc_ping <which> [start|success|fail].
Opt-in via env, exactly like the SMTP2GO / GSC gating: URLs set → ping; unset → clean no-op (keeps local dev + a server-down day silent). Follows the existing feature-gate pattern, not a new always-on dependency.
Best-effort and inert: short curl timeout, || true, runs even during a Claude cooldown (it's non-model). A ping must NEVER affect tick outcome.
Call sites: heartbeat at the top of tick.sh; task start/success around the cascade's work path.
Lockstep gotcha: the env var names in code ↔ host .env must match or the pings silently break (same rule as the SMTP2GO rename).

Done when

healthchecks is up, igor is pinging it, and a deliberately-stopped timer (or a hung tick) raises an alert to you within the grace window — i.e., it would have caught last night. Then we greenlight the code half (add the Agent label / split the follow-up) once the URLs are in .env.

Notes

Two checks = your two alerts; I bundled the dead-man's-switch value into check A so we don't need a third.
Decide during setup whether "starting" pings every tick (simplest, max liveness coverage) or only ticks that begin work — I lean every-tick for A, start/success for B.

## Why Last night the tick timer sat **stopped for ~8.5 hours** and nothing surfaced it — we only caught it because I went looking this morning. We need a **dead-man's switch**: if igor stops ticking (or a tick hangs mid-task), *you* get paged, instead of silence. monit can't cover this — `agent.service` is a per-minute oneshot, not a long-running process monit can watch; a periodic-ping heartbeat is the right tool for a cron job. Set up self-hosted **[healthchecks](https://github.com/healthchecks/healthchecks/)** on the monitoring server and wire two pings out of the tick. ## The two checks (your framing → healthchecks mechanics) **A. "igor is ticking" (heartbeat / dead-man's switch)** — ping at the **start of every tick**. Since ticks fire every minute, this doubles as your "igor starting" signal *and* the liveness alarm: set period ~1 min, grace a few min, and if healthchecks hears nothing it alerts you. **This is the piece that would have caught last night.** **B. "finishing a task"** — ping when a tick completes **real work** (a genuine `# OUTCOME`, not a no-op idle tick). Best as a start→success pair around the work: send `/start` when a `do_*_tick` begins working and the success ping when it returns. Then a tick that **starts a task but never finishes** (a crash/hang — the exact #291 pattern we just fixed) shows up as "started, never completed," and you get per-task duration too. ## Work split **You (this ticket — the server + config):** - Stand up healthchecks on the monitoring box (Docker compose + Postgres + a notification channel — email/ntfy/push to your phone). - Create the two checks; grab their ping URLs; pick period + grace for each. - Drop the ping URL(s) into the host `.env` at `~/.local/share/agent/.env`. **Harness code (a follow-up bot PR once the URLs exist — I can take this):** - `lib/healthcheck.sh` helper, e.g. `hc_ping <which> [start|success|fail]`. - **Opt-in via env**, exactly like the SMTP2GO / GSC gating: URLs set → ping; unset → clean no-op (keeps local dev + a server-down day silent). Follows the existing feature-gate pattern, not a new always-on dependency. - **Best-effort and inert**: short `curl` timeout, `|| true`, runs even during a Claude cooldown (it's non-model). A ping must NEVER affect tick outcome. - Call sites: heartbeat at the top of `tick.sh`; task start/success around the cascade's work path. - Lockstep gotcha: the env var names in code ↔ host `.env` must match or the pings silently break (same rule as the SMTP2GO rename). ## Done when healthchecks is up, igor is pinging it, and **a deliberately-stopped timer (or a hung tick) raises an alert to you within the grace window** — i.e., it would have caught last night. Then we greenlight the code half (add the `Agent` label / split the follow-up) once the URLs are in `.env`. ## Notes - Two checks = your two alerts; I bundled the dead-man's-switch value into check A so we don't need a third. - Decide during setup whether "starting" pings every tick (simplest, max liveness coverage) or only ticks that begin work — I lean every-tick for A, start/success for B.

joshtronic was assigned by igor

2026-06-30 15:12:02 +00:00

igor referenced this issue

2026-07-04 05:16:18 +00:00

feat: healthcheck ping wiring (lib/healthcheck.sh, opt-in) [code half of #297] #339

igor referenced this issue

2026-07-04 19:52:11 +00:00

feat: add healthcheck ping wiring (lib/healthcheck.sh, opt-in) #348

igor referenced this issue

2026-07-04 19:54:40 +00:00

feat: add healthcheck ping wiring (lib/healthcheck.sh, opt-in) #348

igor referenced this issue

2026-07-04 20:05:50 +00:00

feat: add healthcheck ping wiring (lib/healthcheck.sh, opt-in) #348

igor referenced this issue

2026-07-06 19:37:02 +00:00

feat(healthcheck): heartbeat before the security-gate stage (cap tick silence at ~30m, not ~55m) #360

igor commented

2026-07-06 21:38:37 +00:00

Author

Collaborator

Setup note (env var to add) — the code half is done (#348), this is the operator half

The heartbeat wiring is live and no-ops until you set the URL, so just add one env var to ~/.local/share/agent/.env:

HEALTHCHECK_HEARTBEAT_URL="<the ping URL from your healthchecks.io check>"

That's the dead-man's switch — the harness pings it at the top of every tick, and healthchecks.io alerts you if the pings stop.

Optional second check (catches a hung tick, not just a stopped one — pings start/success/fail around the work):

HEALTHCHECK_TASK_URL="<ping URL of a second check>"

healthchecks.io check config

Set Period 30m / Grace 5m (≈35 min tolerance). Why 35 and not tighter: the heartbeat fires per-stage, and the longest single stage (a build, or the security-gate review) is ~30m — so a healthy tick can legitimately be silent up to ~30m. (This is now safe at 35m because the mid-tick heartbeat landed in #360/PR#362; before that a tick could go ~55m and you'd have needed ~60m.) 35m catches a real stoppage fast without false-paging a legit long build.

Notes

No restart needed — tick.sh sources .env at the top of every tick, so it takes effect on the next tick (~1 min).
Keep the URL in .env only (it's a bearer secret; don't commit it).
Once set + you've seen a ping land green, close this issue.

## Setup note (env var to add) — the code half is done (#348), this is the operator half The heartbeat wiring is live and **no-ops until you set the URL**, so just add one env var to `~/.local/share/agent/.env`: ``` HEALTHCHECK_HEARTBEAT_URL="<the ping URL from your healthchecks.io check>" ``` That's the dead-man's switch — the harness pings it at the top of every tick, and healthchecks.io alerts you if the pings stop. **Optional second check** (catches a *hung* tick, not just a stopped one — pings start/success/fail around the work): ``` HEALTHCHECK_TASK_URL="<ping URL of a second check>" ``` ### healthchecks.io check config Set **Period 30m / Grace 5m** (≈35 min tolerance). Why 35 and not tighter: the heartbeat fires per-stage, and the longest single stage (a build, or the security-gate review) is ~30m — so a healthy tick can legitimately be silent up to ~30m. (This is now safe at 35m because the mid-tick heartbeat landed in #360/PR#362; before that a tick could go ~55m and you'd have needed ~60m.) 35m catches a real stoppage fast without false-paging a legit long build. ### Notes - No restart needed — `tick.sh` sources `.env` at the top of every tick, so it takes effect on the next tick (~1 min). - Keep the URL in `.env` only (it's a bearer secret; don't commit it). - Once set + you've seen a ping land green, close this issue.