agent.service tick exited status=28 during recovery sweep #394

New issue

Closed

opened 2026-07-16 05:10:21 +00:00 by igor · 1 comment

igor commented

2026-07-16 05:10:21 +00:00

Collaborator

The 23:46 tick died mid-recovery sweep with a nonzero systemd exit (status=28) and no benign log line explaining it. The tick emitted recovery sweep (igor) and then produced no further output for ~36s before the main process exited abnormally; systemd recorded Failed with result 'exit-code'. Exit 28 has no self-explanation here (commonly ENOSPC/No space left on device or a curl operation timeout), so the root cause is unclear. The next tick (23:48) recovered, so this is a single-window crash rather than a stuck loop, but an unexplained nonzero exit of the harness itself is worth a ticket.

Jul 15 23:46:47 igor.sherver.org tick.sh[1958452]: [agent] recovery sweep (igor)
Jul 15 23:47:23 igor.sherver.org systemd[798]: agent.service: Main process exited, code=exited, status=28/n/a
Jul 15 23:47:23 igor.sherver.org systemd[798]: agent.service: Failed with result 'exit-code'.
Jul 15 23:47:23 igor.sherver.org systemd[798]: Failed to start agent.service - Agent -- one tick: claim and work one Forgejo issue across known projects.

Fixed = the recovery sweep either completes or fails with a logged, human-readable reason; the harness should trap the failing sub-step (likely a curl/disk op) and exit 0 for the tick or emit an explicit [agent] recovery: <error> line, so an unattributed status=28 never reaches systemd.

service: agent.service
severity: medium
window: 2026-07-15 23:00-00:00 (filed by the hourly logwatch pass)

The 23:46 tick died mid-`recovery sweep` with a nonzero systemd exit (`status=28`) and no benign log line explaining it. The tick emitted `recovery sweep (igor)` and then produced no further output for ~36s before the main process exited abnormally; systemd recorded `Failed with result 'exit-code'`. Exit 28 has no self-explanation here (commonly ENOSPC/`No space left on device` or a curl operation timeout), so the root cause is unclear. The next tick (23:48) recovered, so this is a single-window crash rather than a stuck loop, but an unexplained nonzero exit of the harness itself is worth a ticket. ``` Jul 15 23:46:47 igor.sherver.org tick.sh[1958452]: [agent] recovery sweep (igor) Jul 15 23:47:23 igor.sherver.org systemd[798]: agent.service: Main process exited, code=exited, status=28/n/a Jul 15 23:47:23 igor.sherver.org systemd[798]: agent.service: Failed with result 'exit-code'. Jul 15 23:47:23 igor.sherver.org systemd[798]: Failed to start agent.service - Agent -- one tick: claim and work one Forgejo issue across known projects. ``` Fixed = the recovery sweep either completes or fails with a logged, human-readable reason; the harness should trap the failing sub-step (likely a `curl`/disk op) and exit 0 for the tick or emit an explicit `[agent] recovery: <error>` line, so an unattributed `status=28` never reaches systemd. --- service: agent.service severity: medium window: 2026-07-15 23:00-00:00 (filed by the hourly logwatch pass)

joshtronic was assigned by igor

2026-07-16 05:10:22 +00:00

igor added spent time

2026-07-16 05:10:22 +00:00

33 seconds

igor commented

2026-07-16 13:59:57 +00:00

Author

Collaborator

Duplicate of #395 — same 23:47 status=28 incident. Root-caused there: a brief git.sherver.org blip hung an _fj Forgejo API curl ~30s (--max-time 30, no --connect-timeout) → curl exit 28 → errexit abort → the cleanup() trap fired a false task fail page; the next tick self-recovered.

Fixed in #396 (fail-fast --connect-timeout 5 + --max-time 30→15). Your suggested fix here — trap the failing sub-step, emit an explicit [agent] recovery: <error> line, and let the tick exit 0 — is captured as the noted follow-up in #395 (make the idempotent recovery-sweep calls best-effort). Independent confirmation of the same diagnosis, which is reassuring.

Closing as a duplicate; track the fix on #396.

Duplicate of #395 — same 23:47 `status=28` incident. Root-caused there: a brief git.sherver.org blip hung an `_fj` Forgejo API curl ~30s (`--max-time 30`, no `--connect-timeout`) → curl exit 28 → errexit abort → the `cleanup()` trap fired a false `task fail` page; the next tick self-recovered. **Fixed in #396** (fail-fast `--connect-timeout 5` + `--max-time` 30→15). Your suggested fix here — trap the failing sub-step, emit an explicit `[agent] recovery: <error>` line, and let the tick exit 0 — is captured as the noted follow-up in #395 (make the idempotent recovery-sweep calls best-effort). Independent confirmation of the same diagnosis, which is reassuring. Closing as a duplicate; track the fix on #396.

igor

2026-07-16 13:59:57 +00:00