Forgejo API curls lack --connect-timeout: a brief git.sherver.org blip wedges a tick ~30s and false-trips the healthcheck #395

New issue

Closed

opened 2026-07-16 13:36:23 +00:00 by igor · 0 comments

igor commented

2026-07-16 13:36:23 +00:00

Collaborator

Symptom

A healthchecks.io task-fail alert paged the operator ~2026-07-15 23:47. The harness was actually healthy — it self-recovered on the very next tick — but the page was real: one tick failed with status=28.

Diagnosis

git.sherver.org had a brief (~few-second) blip. During it, a Forgejo API curl in the early cascade hung until its --max-time 30 expired → curl exit 28 (timeout). Under the tick's set -euo pipefail that aborted the tick (rc=28); the cleanup() EXIT trap saw HC_TASK_STARTED=1 + nonzero rc and fired hc_ping task fail (tick.sh ~170) → the check flipped down → alert. The next tick succeeded and re-pinged success (~2 min later) → auto-recovered.

Rare — 2 failed ticks in 5 days (this exit-28 + an unrelated exit-3). But a few-second server blip should not wedge a tick ~30s or page the operator.

Root cause

lib/forgejo.sh's curl wrappers use --max-time 30 (and 60 for uploads) with no --connect-timeout. A briefly-unreachable server stalls a call for the full max-time before failing, instead of failing fast (or riding through a sub-second blip). Call volume is not the issue: validation git-fetches are cached 15 min/repo (VALIDATION_COOLDOWN_SECS=900), and a tick makes only ~10-30 small API calls.

Fix (this PR)

Add --connect-timeout 5 and tighten --max-time 30 → 15 across the Forgejo curl paths (_fj, _forgejo_post_reviewers; the image upload keeps --max-time 60 but gains the connect-timeout). A blip now fails (or rides through) in ~5s instead of ~30s — keeps the agent moving. Values hardcoded (one operator). Test shims curl and asserts _fj passes the fail-fast flags.

Follow-up (noted, not in this PR)

Even with fast timeouts, an unguarded best-effort caller (e.g. the recovery sweep's forgejo_comment / forgejo_unassign_all) still propagates a transient failure fatally, so a blip can still page the operator — just faster. Worth making those idempotent recovery calls best-effort (log + continue) so a transient blip doesn't page at all. The co-located basename reaper bug (#392) rides this same path — fixed in PR #393.

## Symptom A healthchecks.io **task-fail** alert paged the operator ~2026-07-15 23:47. The harness was actually healthy — it self-recovered on the very next tick — but the page was real: one tick failed with `status=28`. ## Diagnosis git.sherver.org had a brief (~few-second) blip. During it, a Forgejo API `curl` in the early cascade hung until its **`--max-time 30`** expired → **curl exit 28** (timeout). Under the tick's `set -euo pipefail` that aborted the tick (rc=28); the `cleanup()` EXIT trap saw `HC_TASK_STARTED=1` + nonzero rc and fired `hc_ping task fail` (tick.sh ~170) → the check flipped down → alert. The next tick succeeded and re-pinged success (~2 min later) → auto-recovered. Rare — **2 failed ticks in 5 days** (this exit-28 + an unrelated exit-3). But a few-second server blip should not wedge a tick ~30s or page the operator. ## Root cause `lib/forgejo.sh`'s curl wrappers use `--max-time 30` (and `60` for uploads) with **no `--connect-timeout`**. A briefly-unreachable server stalls a call for the full max-time before failing, instead of failing fast (or riding through a sub-second blip). Call *volume* is not the issue: validation git-fetches are cached 15 min/repo (`VALIDATION_COOLDOWN_SECS=900`), and a tick makes only ~10-30 small API calls. ## Fix (this PR) Add `--connect-timeout 5` and tighten `--max-time` 30 → 15 across the Forgejo curl paths (`_fj`, `_forgejo_post_reviewers`; the image upload keeps `--max-time 60` but gains the connect-timeout). A blip now fails (or rides through) in ~5s instead of ~30s — keeps the agent moving. Values hardcoded (one operator). Test shims `curl` and asserts `_fj` passes the fail-fast flags. ## Follow-up (noted, not in this PR) Even with fast timeouts, an unguarded best-effort caller (e.g. the recovery sweep's `forgejo_comment` / `forgejo_unassign_all`) still propagates a transient failure fatally, so a blip can still page the operator — just faster. Worth making those idempotent recovery calls best-effort (log + continue) so a transient blip doesn't page at all. The co-located `basename` reaper bug (#392) rides this same path — fixed in PR #393.

igor added the

Kind/Bug

label

2026-07-16 13:36:23 +00:00

igor referenced this issue from a commit

2026-07-16 13:38:57 +00:00

Fail-fast Forgejo timeouts: add --connect-timeout, tighten --max-time

igor referenced this issue from a pull request that will close it,

2026-07-16 13:39:09 +00:00

Fail-fast Forgejo timeouts: --connect-timeout + tighter --max-time #396

igor referenced this issue

2026-07-16 13:41:37 +00:00