agent.service exited 1 mid-rework on porksicle.com#89 (822M peak, work lost and redone) #272

New issue

Closed

opened 2026-06-27 06:11:10 +00:00 by igor · 1 comment

igor commented

2026-06-27 06:11:10 +00:00

Collaborator

The 00:39 tick claimed the binding rework for joshtronic/porksicle.com#89, invoked claude at 00:39:57, and ran ~14 min of heavy work in the worktree (npm install, generating thumb.png/og.png, npm test, npm run verify with Playwright/Chromium, security review). Mid-tool-result the harness exited nonzero with no [agent] claude exited / time logged completion markers and no benign explanation. Memory peak was 822.5M -- far above the ~150-360M of every other tick this hour, suggesting the Chromium/Playwright path under the unit blew its budget. The crash discarded all the rework; the 00:55 tick re-opened the same PR and began regenerating the identical thumbnail from scratch (still running at hour close), so the expensive operation is on track to repeat -- and re-crash -- each round.

Jun 27 00:53:45 igor tick.sh[1224857]: [tool_result]
Jun 27 00:53:46 igor systemd[785]: agent.service: Main process exited, code=exited, status=1/FAILURE
Jun 27 00:53:46 igor systemd[785]: agent.service: Failed with result 'exit-code'.
Jun 27 00:53:46 igor systemd[785]: Failed to start agent.service - Agent -- one tick: claim and work one Forgejo issue across known projects.
Jun 27 00:53:46 igor systemd[785]: agent.service: Consumed 1min 35.748s CPU time, 822.5M memory peak.

Followed by the from-scratch redo:

Jun 27 00:56:06 igor tick.sh[1228304]: [agent] PR-review: joshtronic/porksicle.com#89 -- reopening (reassigned back to bot)
Jun 27 00:56:09 igor tick.sh[1229114]: HEAD is now at c3bad2c chore(hog-and-seek): drop the ?test introspection hook before shipping

'Fixed' = the rework tick either completes (commits + requests review) or fails gracefully with a logged error instead of a raw status=1/FAILURE; identify what threw at 00:53:46 (likely the Playwright/Chromium npm run verify under the unit's memory cap) and either bound its memory or run it outside the per-minute tick so a single review round can't crash and indefinitely re-do the same work.

service: agent.service
severity: medium
window: 2026-06-27 00:00-01:00 (filed by the hourly logwatch pass)

The 00:39 tick claimed the binding rework for `joshtronic/porksicle.com#89`, invoked claude at 00:39:57, and ran ~14 min of heavy work in the worktree (npm install, generating `thumb.png`/`og.png`, `npm test`, `npm run verify` with Playwright/Chromium, security review). Mid-tool-result the harness exited nonzero with no `[agent] claude exited` / `time logged` completion markers and no benign explanation. Memory peak was 822.5M -- far above the ~150-360M of every other tick this hour, suggesting the Chromium/Playwright path under the unit blew its budget. The crash discarded all the rework; the 00:55 tick re-opened the same PR and began regenerating the identical thumbnail from scratch (still running at hour close), so the expensive operation is on track to repeat -- and re-crash -- each round. ``` Jun 27 00:53:45 igor tick.sh[1224857]: [tool_result] Jun 27 00:53:46 igor systemd[785]: agent.service: Main process exited, code=exited, status=1/FAILURE Jun 27 00:53:46 igor systemd[785]: agent.service: Failed with result 'exit-code'. Jun 27 00:53:46 igor systemd[785]: Failed to start agent.service - Agent -- one tick: claim and work one Forgejo issue across known projects. Jun 27 00:53:46 igor systemd[785]: agent.service: Consumed 1min 35.748s CPU time, 822.5M memory peak. ``` Followed by the from-scratch redo: ``` Jun 27 00:56:06 igor tick.sh[1228304]: [agent] PR-review: joshtronic/porksicle.com#89 -- reopening (reassigned back to bot) Jun 27 00:56:09 igor tick.sh[1229114]: HEAD is now at c3bad2c chore(hog-and-seek): drop the ?test introspection hook before shipping ``` 'Fixed' = the rework tick either completes (commits + requests review) or fails gracefully with a logged error instead of a raw `status=1/FAILURE`; identify what threw at 00:53:46 (likely the Playwright/Chromium `npm run verify` under the unit's memory cap) and either bound its memory or run it outside the per-minute tick so a single review round can't crash and indefinitely re-do the same work. --- service: agent.service severity: medium window: 2026-06-27 00:00-01:00 (filed by the hourly logwatch pass)

joshtronic was assigned by igor

2026-06-27 06:11:10 +00:00

igor added spent time

2026-06-27 06:11:11 +00:00

54 seconds

igor referenced this issue from a pull request that will close it,

2026-06-27 13:33:30 +00:00

fix(claude): guard cost/health bookkeeping so a mid-stream claude exit can't crash the tick (#272) #276

igor closed this issue

2026-06-27 13:33:31 +00:00

igor referenced this issue from a commit

2026-06-27 13:33:32 +00:00

fix(claude): bookkeeping can't crash the tick on a mid-stream claude exit

igor commented

2026-06-27 13:34:47 +00:00

Author

Collaborator

Closing in favor of #276, which fixes the real cause. This was not OOM — 31 GiB RAM (27 free, swap untouched), exit code 1 (an OOM-kill is SIGKILL → 137), and no kernel OOM-killer entries; the 822M peak was just Chromium under Playwright. The actual bug: claude exited nonzero mid-stream, truncating the cost JSONL, so cost_record_cli failed under a re-enabled set -e and raised a raw status=1 that bypassed the caller’s exit handling. Fixed in #276.

Closing in favor of #276, which fixes the real cause. **This was not OOM** — 31 GiB RAM (27 free, swap untouched), exit code 1 (an OOM-kill is SIGKILL → 137), and no kernel OOM-killer entries; the 822M peak was just Chromium under Playwright. The actual bug: claude exited nonzero mid-stream, truncating the cost JSONL, so cost_record_cli failed under a re-enabled set -e and raised a raw status=1 that bypassed the caller’s exit handling. Fixed in #276.

igor referenced this issue

2026-06-27 13:37:12 +00:00

fix(claude): guard cost/health bookkeeping so a mid-stream claude exit can't crash the tick (#272) #276

joshtronic referenced this issue from a commit

2026-06-27 13:41:15 +00:00

Merge pull request 'fix(claude): guard cost/health bookkeeping so a mid-stream claude exit can't crash the tick (#272)' (#276) from fix/claude-cost-record-crash into master

igor referenced this issue from a commit

2026-06-28 15:29:06 +00:00

feat(crashlog): preserve the claude stream when a tick dies mid-call (observe #279)

igor referenced this issue

2026-06-28 15:29:21 +00:00

feat(crashlog): preserve the claude stream when a tick dies mid-call (observe #279) #280