fix(tick): install cleanup trap early + break the rework crash-loop (igor#291) #294

Merged

joshtronic merged 1 commit from feat/crash-fix into master

2026-06-30 13:18:57 +00:00

igor commented

2026-06-30 04:19:39 +00:00

Collaborator

Fixes the crash-loop from logwatch igor#291 (#1+#2 of the night shift). The tick crashed (systemd status=1, not OOM-137 -- 3.1G on a 31GiB host) mid-PR-rework on a Node repo with a heavy in-worktree test suite, then re-attempted every tick forever, starving the cascade.

Two root causes, two fixes:

The crashlog gap -- #280s capture never fired because trap cleanup EXIT was installed after the early worktree-creating stages (maintenance, PR-rework, the claude call). A crash there ran with no handler. Moved cleanup() + the trap to ~line 143-160 (right after the bot-identity exports) so every stage is covered; cleanup is a safe no-op on early/clean exits.
The starvation loop -- a crash-safe rework counter (.review[key].rework_crashes): stamp before the work, reset on a clean return, and after 2 crashes escalate to you (comment + unassign bot + assign reviewer) instead of looping.

Honest note: the exact uncaught set -e line is elusive from static reading (visible post-call code is guarded). Fix #1 means the next mid-rework crash gets preserved to crash-logs/, handing us the stream to pin it -- fix #2 makes that non-urgent. No containment lever added (memory is not the cause; cgroup/systemd are off-limits).

Verified: bash -n clean, check-sync PASS, test-crashlog green. Reviewed by hand independently. No reviewer.

Fixes the crash-loop from logwatch **igor#291** (#1+#2 of the night shift). The tick crashed (systemd `status=1`, **not** OOM-137 -- 3.1G on a 31GiB host) mid-PR-rework on a Node repo with a heavy in-worktree test suite, then re-attempted every tick forever, starving the cascade. **Two root causes, two fixes:** 1. **The crashlog gap** -- `#280`s capture never fired because `trap cleanup EXIT` was installed *after* the early worktree-creating stages (maintenance, PR-rework, the claude call). A crash there ran with no handler. **Moved `cleanup()` + the trap to ~line 143-160** (right after the bot-identity exports) so every stage is covered; cleanup is a safe no-op on early/clean exits. 2. **The starvation loop** -- a crash-safe rework counter (`.review[key].rework_crashes`): stamp before the work, reset on a clean return, and after **2** crashes escalate to you (comment + unassign bot + assign reviewer) instead of looping. Honest note: the *exact* uncaught `set -e` line is elusive from static reading (visible post-call code is guarded). Fix #1 means the next mid-rework crash gets **preserved to `crash-logs/`**, handing us the stream to pin it -- fix #2 makes that non-urgent. No containment lever added (memory is not the cause; cgroup/systemd are off-limits). Verified: `bash -n` clean, check-sync PASS, test-crashlog green. Reviewed by hand independently. No reviewer.

igor added 1 commit

2026-06-30 04:19:39 +00:00

fix(tick): install cleanup trap early + break the rework crash-loop (igor#291)

Lint / check-sync (pull_request) Successful in 5s

Details

Lint / check-sync (push) Successful in 4s

Details

4fbd031cc1

The tick crashed (systemd status=1, NOT OOM-137) mid-PR-rework on a Node repo whose
definition-of-done runs a heavy in-worktree test suite, then re-attempted every tick
forever -- starving the downstream cascade. Two root causes:

1. Crashlog gap (#280 couldn't catch it): 'trap cleanup EXIT' was installed AFTER the
   early worktree-creating stages (maintenance ~1360, PR-rework ~2970, claude ~3227).
   A crash there fired with NO trap installed -- crashlog_preserve never ran, the
   worktree leaked. Moved cleanup() + the trap + WORKTREE/PR_WORKTREE init to ~line
   143-160, right after the bot-identity exports, so every worktree-creating stage is
   covered. cleanup() is a safe no-op on early/clean exits (guarded vars,
   short-circuits, preserves $rc).

2. Crash-loop: a rework dying mid-run was reopened+reassigned every tick. Added a
   crash-safe counter (.review[key].rework_crashes, mirroring rework_rounds): stamp
   the attempt BEFORE the work, reset only on a clean return past the claude call, and
   after REWORK_CRASH_CAP=2 crashes escalate to the human (comment + unassign bot +
   assign reviewer) instead of re-attempting.

The exact uncaught set -e abort is elusive from static reading (the visible post-call
code is guarded); fix #1 ensures the NEXT mid-rework crash is preserved to crash-logs/
-- handing us the stream to pin the line -- and fix #2 makes that non-urgent by
breaking the loop.

bash -n clean; check-sync PASS; test-crashlog green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011KXPafSYJPY6XkWHHk1Lfs