agent.service tick exits 28 during recovery sweep with no error output #424

New issue

Closed

opened 2026-07-25 06:02:34 +00:00 by igor · 0 comments

igor commented

2026-07-25 06:02:34 +00:00

Collaborator

Diagnosis

The 00:32 tick died mid-run: it logged recovery sweep (igor) at 00:32:26 and then the main process exited status=28 eleven seconds later, with no error line, no stack trace, and no partial output from the sweep. systemd recorded a hard failure (Failed with result 'exit-code'), so this tick abandoned its window entirely — no maintenance/seo/logwatch/claim phases ran for that minute.

Exit 28 is suspicious as a leaked status rather than a deliberate one: it matches curl exit 28 (operation timeout) and ENOSPC/sqlite full-disk conditions. Most likely a command inside the recovery sweep is failing under set -e and its exit code is propagating out of tick.sh unmapped, so the failure surfaces as an opaque systemd unit failure instead of a logged, retried step. Every other tick in the hour completed normally, so this is not a persistent outage — but a silent nonzero exit with zero diagnostics is a harness bug, not a blip, and it will recur invisibly.

Evidence

Jul 25 00:32:01 igor.sherver.org systemd[805]: Starting agent.service - Agent -- one tick: claim and work one Forgejo issue across known projects...
Jul 25 00:32:10 igor.sherver.org tick.sh[1440854]: [agent] validation: 0 pass, 15 cached, 4 not-ready, 0 indeterminate
Jul 25 00:32:26 igor.sherver.org tick.sh[1440854]: [agent] recovery sweep (igor)
Jul 25 00:32:37 igor.sherver.org systemd[805]: agent.service: Main process exited, code=exited, status=28/n/a
Jul 25 00:32:37 igor.sherver.org systemd[805]: agent.service: Failed with result 'exit-code'
Jul 25 00:32:37 igor.sherver.org systemd[805]: Failed to start agent.service - Agent -- one tick: claim and work one Forgejo issue across known projects.
Jul 25 00:32:37 igor.sherver.org systemd[805]: agent.service: Consumed 6.039s CPU time, 11.9M memory peak.

Contrast with the healthy neighbours, which both reach no claimable work -- idle and Finished agent.service:

Jul 25 00:30:18 igor.sherver.org systemd[805]: Finished agent.service - Agent -- one tick: claim and work one Forgejo issue across known projects.
Jul 25 00:35:38 igor.sherver.org systemd[805]: Finished agent.service - Agent -- one tick: claim and work one Forgejo issue across known projects.

What "fixed" looks like

The recovery sweep wraps its fallible subcommands (network calls, git/clone operations, sqlite writes) so a failure logs a [agent] recovery: <step> failed rc=<n> line naming the step and the underlying command before anything exits.
A transient sub-step failure inside the sweep does not abort the whole tick; the tick continues to the remaining phases (or exits 0 after logging) so a single sweep hiccup doesn't silently forfeit a window.
If the tick genuinely must abort, it exits with a mapped, documented status and prints a one-line reason, so status=28/n/a never appears without a matching tick.sh log line.

service: agent.service
severity: medium
window: 2026-07-25 00:00-01:00 (filed by the hourly logwatch pass)

## Diagnosis The 00:32 tick died mid-run: it logged `recovery sweep (igor)` at 00:32:26 and then the main process exited `status=28` eleven seconds later, with no error line, no stack trace, and no partial output from the sweep. systemd recorded a hard failure (`Failed with result 'exit-code'`), so this tick abandoned its window entirely — no maintenance/seo/logwatch/claim phases ran for that minute. Exit 28 is suspicious as a *leaked* status rather than a deliberate one: it matches `curl` exit 28 (operation timeout) and `ENOSPC`/`sqlite` full-disk conditions. Most likely a command inside the recovery sweep is failing under `set -e` and its exit code is propagating out of `tick.sh` unmapped, so the failure surfaces as an opaque systemd unit failure instead of a logged, retried step. Every other tick in the hour completed normally, so this is not a persistent outage — but a silent nonzero exit with zero diagnostics is a harness bug, not a blip, and it will recur invisibly. ## Evidence ``` Jul 25 00:32:01 igor.sherver.org systemd[805]: Starting agent.service - Agent -- one tick: claim and work one Forgejo issue across known projects... Jul 25 00:32:10 igor.sherver.org tick.sh[1440854]: [agent] validation: 0 pass, 15 cached, 4 not-ready, 0 indeterminate Jul 25 00:32:26 igor.sherver.org tick.sh[1440854]: [agent] recovery sweep (igor) Jul 25 00:32:37 igor.sherver.org systemd[805]: agent.service: Main process exited, code=exited, status=28/n/a Jul 25 00:32:37 igor.sherver.org systemd[805]: agent.service: Failed with result 'exit-code' Jul 25 00:32:37 igor.sherver.org systemd[805]: Failed to start agent.service - Agent -- one tick: claim and work one Forgejo issue across known projects. Jul 25 00:32:37 igor.sherver.org systemd[805]: agent.service: Consumed 6.039s CPU time, 11.9M memory peak. ``` Contrast with the healthy neighbours, which both reach `no claimable work -- idle` and `Finished agent.service`: ``` Jul 25 00:30:18 igor.sherver.org systemd[805]: Finished agent.service - Agent -- one tick: claim and work one Forgejo issue across known projects. Jul 25 00:35:38 igor.sherver.org systemd[805]: Finished agent.service - Agent -- one tick: claim and work one Forgejo issue across known projects. ``` ## What "fixed" looks like - The recovery sweep wraps its fallible subcommands (network calls, git/clone operations, sqlite writes) so a failure logs a `[agent] recovery: <step> failed rc=<n>` line naming the step and the underlying command before anything exits. - A transient sub-step failure inside the sweep does not abort the whole tick; the tick continues to the remaining phases (or exits 0 after logging) so a single sweep hiccup doesn't silently forfeit a window. - If the tick genuinely must abort, it exits with a mapped, documented status and prints a one-line reason, so `status=28/n/a` never appears without a matching `tick.sh` log line. --- service: agent.service severity: medium window: 2026-07-25 00:00-01:00 (filed by the hourly logwatch pass)