Preserve claude's raw stream on a nonzero exit, so a claude-bomb is diagnosable #326

New issue

Closed

opened 2026-07-02 21:07:23 +00:00 by igor · 1 comment

igor commented

2026-07-02 21:07:23 +00:00

Collaborator

Blind spot

When claude exits nonzero on its own (rc=${PIPESTATUS[0]} in claude_run_with_cost), the harness handles it correctly — ship-safety refuses the partial worktree and re-queues. But the raw claude stream ($scratch/claude-stream.jsonl) lives in the worktree's .agent/ scratch, which is torn down with the worktree. crashlog_preserve only fires when the tick itself dies before claude_run_with_cost returns (the #279 signature). On a clean rc=1 return there is nothing preserved — so there's no forensic trail for why claude exited nonzero.

Concrete cost

vps-showdown#41 bombed twice — claude exited 1 after ballooning to 3.4–4.7 GB (~350× a normal tick), producing no committable work. The root cause is only inferable (likely a Node/V8 heap blowup from trying to build a generated site), because the stream was already gone by the time anyone looked. Every "claude exited N" is currently un-diagnosable after the fact.

Fix

On a nonzero claude exit (rc ≠ 0, distinct from a clean timeout), preserve the raw stream + display log to a durable location — same mechanism as crashlog_preserve — keyed by call-site + timestamp, size-capped and rotated so it can't grow unbounded. Then a bomb leaves a readable post-mortem instead of a guess.

Notes

Delicate: touches claude_run_with_cost / the crashlog path. Should be done carefully with a bin/test-*.sh case, not a fast mechanical edit.
This is purely observability — the harness handled the #41 bomb correctly (ship-safety + 2-strike → Status/Blocked). No behavior bug here.

## Blind spot When claude exits nonzero on its own (`rc=${PIPESTATUS[0]}` in `claude_run_with_cost`), the harness handles it correctly — ship-safety refuses the partial worktree and re-queues. But the raw claude stream (`$scratch/claude-stream.jsonl`) lives in the worktree's `.agent/` scratch, which is torn down with the worktree. `crashlog_preserve` only fires when the **tick itself dies** before `claude_run_with_cost` returns (the #279 signature). On a clean `rc=1` return there is nothing preserved — so there's **no forensic trail for why claude exited nonzero**. ## Concrete cost `vps-showdown#41` bombed twice — claude exited 1 after ballooning to **3.4–4.7 GB** (~350× a normal tick), producing no committable work. The root cause is only *inferable* (likely a Node/V8 heap blowup from trying to build a generated site), because the stream was already gone by the time anyone looked. Every "claude exited N" is currently un-diagnosable after the fact. ## Fix On a nonzero claude exit (rc ≠ 0, distinct from a clean timeout), preserve the raw stream + display log to a durable location — same mechanism as `crashlog_preserve` — keyed by call-site + timestamp, size-capped and rotated so it can't grow unbounded. Then a bomb leaves a readable post-mortem instead of a guess. ## Notes - **Delicate**: touches `claude_run_with_cost` / the crashlog path. Should be done carefully with a `bin/test-*.sh` case, not a fast mechanical edit. - This is purely observability — the harness handled the #41 bomb correctly (ship-safety + 2-strike → `Status/Blocked`). No behavior bug here.

joshtronic was assigned by igor

2026-07-02 21:07:23 +00:00

igor commented

2026-07-02 21:19:05 +00:00

Author

Collaborator

Concrete symptom that motivated this: logwatch #325 — claude exited 1 twice on vps-showdown#41 (4.7 GB peak) and the crash reason was unknowable because the stream wasn't preserved. This issue is the visibility half of closing that gap; the scope half is vps-showdown#46 (re-scoped, build-free). Closing #325 in favor of the two.

Concrete symptom that motivated this: logwatch **#325** — claude exited 1 twice on vps-showdown#41 (4.7 GB peak) and the crash reason was **unknowable** because the stream wasn't preserved. This issue is the *visibility* half of closing that gap; the *scope* half is vps-showdown#46 (re-scoped, build-free). Closing #325 in favor of the two.

igor referenced this issue from a commit

2026-07-03 01:20:58 +00:00

fix(observability): preserve claude's stream on a nonzero exit (igor#326)

igor referenced this issue from a pull request that will close it,

2026-07-03 01:21:09 +00:00

fix(observability): preserve claude's stream on a nonzero exit (closes #326) #327

joshtronic closed this issue

2026-07-03 02:06:10 +00:00

joshtronic referenced this issue from a commit

2026-07-03 02:06:12 +00:00

Merge pull request 'fix(observability): preserve claude's stream on a nonzero exit (closes #326)' (#327) from fix/preserve-claude-stream-on-nonzero into master