agent.service exits 1 at security-review Skill, abandoning PR-review rework mid-tick

igor commented

2026-06-28 04:26:48 +00:00

Collaborator

Diagnosis

Twice in this hour the PR-review rework tick for joshtronic/porksicle.com#91 died abruptly with status=1/FAILURE at the moment claude invoked the security-review [tool: Skill] step. In both cases there is no [agent] claude exited ... line and no harness-commit — the tick just dies, discarding the completed rework (the 22:17 tick had already redone the 6-level reconciliation; the 22:49 tick had finished the R-key restart fix + Playwright verification). Compare the healthy ticks (22:11, 22:42) where the same security review runs to completion and logs [agent] claude exited 0 after ...s before committing.

The crash is non-deterministic (the 22:33 tick passed through the same Skill cleanly) but recurred twice in one hour. Net effect: PR #91 is burning full claude invocations (500s+ each) that get thrown away, forcing the harness to re-review from scratch each tick.

Evidence

Jun 27 22:31:16 igor tick.sh[2012624]: [tool: Skill]
Jun 27 22:32:11 igor tick.sh[2012624]: [tool_result]
Jun 27 22:32:11 igor systemd[785]: agent.service: Main process exited, code=exited, status=1/FAILURE
Jun 27 22:32:11 igor systemd[785]: agent.service: Failed with result 'exit-code'.

Jun 27 22:57:14 igor tick.sh[2020010]: [tool: Skill]
Jun 27 22:57:14 igor tick.sh[2020010]: [tool_result]
Jun 27 22:57:14 igor systemd[785]: agent.service: Main process exited, code=exited, status=1/FAILURE
Jun 27 22:57:14 igor systemd[785]: agent.service: Failed with result 'exit-code'.

The recent commit fix(claude): bookkeeping can't crash the tick on a mid-stream claude exit targets an adjacent crash, but the failure recurs twice in this window, so that guard does not cover the security-review/Skill exit path.

Fixed looks like

Invoking the security-review Skill can no longer take the tick down with exit 1: claude either completes the review and the harness commits/re-requests review, or any mid-stream sub-process exit is caught and logged ([agent] claude exited N) and the tick exits 0 without abandoning the worktree.

service: agent.service
severity: medium
window: 2026-06-27 22:00-23:00 (filed by the hourly logwatch pass)

## Diagnosis Twice in this hour the PR-review rework tick for `joshtronic/porksicle.com#91` died abruptly with `status=1/FAILURE` at the moment claude invoked the security-review `[tool: Skill]` step. In both cases there is **no** `[agent] claude exited ...` line and **no** harness-commit — the tick just dies, discarding the completed rework (the 22:17 tick had already redone the 6-level reconciliation; the 22:49 tick had finished the R-key restart fix + Playwright verification). Compare the healthy ticks (22:11, 22:42) where the same security review runs to completion and logs `[agent] claude exited 0 after ...s` before committing. The crash is non-deterministic (the 22:33 tick passed through the same Skill cleanly) but recurred twice in one hour. Net effect: PR #91 is burning full claude invocations (500s+ each) that get thrown away, forcing the harness to re-review from scratch each tick. ## Evidence ``` Jun 27 22:31:16 igor tick.sh[2012624]: [tool: Skill] Jun 27 22:32:11 igor tick.sh[2012624]: [tool_result] Jun 27 22:32:11 igor systemd[785]: agent.service: Main process exited, code=exited, status=1/FAILURE Jun 27 22:32:11 igor systemd[785]: agent.service: Failed with result 'exit-code'. ``` ``` Jun 27 22:57:14 igor tick.sh[2020010]: [tool: Skill] Jun 27 22:57:14 igor tick.sh[2020010]: [tool_result] Jun 27 22:57:14 igor systemd[785]: agent.service: Main process exited, code=exited, status=1/FAILURE Jun 27 22:57:14 igor systemd[785]: agent.service: Failed with result 'exit-code'. ``` The recent commit `fix(claude): bookkeeping can't crash the tick on a mid-stream claude exit` targets an adjacent crash, but the failure recurs twice *in this window*, so that guard does not cover the security-review/Skill exit path. ## Fixed looks like Invoking the security-review Skill can no longer take the tick down with exit 1: claude either completes the review and the harness commits/re-requests review, or any mid-stream sub-process exit is caught and logged (`[agent] claude exited N`) and the tick exits 0 without abandoning the worktree. --- service: agent.service severity: medium window: 2026-06-27 22:00-23:00 (filed by the hourly logwatch pass)

joshtronic was assigned by igor

2026-06-28 04:26:49 +00:00

igor added spent time

2026-06-28 04:26:49 +00:00

1 minute 28 seconds

igor referenced this issue from a commit

2026-06-28 15:29:06 +00:00

feat(crashlog): preserve the claude stream when a tick dies mid-call (observe #279)

igor referenced this issue

2026-06-28 15:29:21 +00:00

feat(crashlog): preserve the claude stream when a tick dies mid-call (observe #279) #280

igor commented

2026-06-28 15:29:41 +00:00

Author

Collaborator

Investigated but deliberately NOT patched with a guess — observability added in #280 instead. Key finding: the post-pipe bookkeeping in claude_run_with_cost is provably abort-proof (cost/health are guarded; classify and record_failure both return 0), so the death is in the claude / security-review-Skill PROCESS, not the harness. That means the actual cause is in claude’s stderr, which the harness currently throws away on a crash. #280 preserves the raw claude stream to ~/.local/state/agent/crash-logs/ whenever a tick dies mid-call, so the next occurrence will show the real cause. Leaving this open until that lands a diagnosis.

Investigated but deliberately NOT patched with a guess — observability added in #280 instead. Key finding: the post-pipe bookkeeping in `claude_run_with_cost` is provably abort-proof (cost/health are guarded; `classify` and `record_failure` both return 0), so the death is in the claude / security-review-Skill PROCESS, not the harness. That means the actual cause is in claude’s stderr, which the harness currently throws away on a crash. #280 preserves the raw claude stream to `~/.local/state/agent/crash-logs/` whenever a tick dies mid-call, so the next occurrence will show the real cause. Leaving this open until that lands a diagnosis.

igor referenced this issue

2026-06-28 15:31:57 +00:00

feat(crashlog): preserve the claude stream when a tick dies mid-call (observe #279) #280

joshtronic referenced this issue from a commit

2026-06-28 15:52:23 +00:00

Merge pull request 'feat(crashlog): preserve the claude stream when a tick dies mid-call (observe #279)' (#280) from feat/preserve-claude-stream-on-crash into master

igor commented

2026-06-29 18:09:54 +00:00

Author

Collaborator

Once-over -- closing. The crash last fired Jun 27 (3x that night, all on the porksicle#91 rework hitting the security-review Skill); zero recurrences since (Jun 28-29 clean), and the triggering rework has resolved. Root stays undiagnosed, but it is a claude / security-review-Skill PROCESS death, not the harness bookkeeping (which I verified is abort-proof -- cost/health are guarded, classify and record_failure return 0). #280 is now live and preserves the claude stream to ~/.local/state/agent/crash-logs/ on the next mid-call death, so if this returns we will have the actual cause and logwatch will re-file. No actionable work is pending, so closing as not-currently-reproducing with the observability as the safety net.

igor closed this issue

2026-06-29 18:09:54 +00:00

igor referenced this issue

2026-07-02 21:07:23 +00:00

Preserve claude's raw stream on a nonzero exit, so a claude-bomb is diagnosable #326

Rows
Columns

agent.service exits 1 at security-review Skill, abandoning PR-review rework mid-tick #279

Diagnosis

Evidence

Fixed looks like