fix: security gate no-verdict is deterministic on gate-describing diffs -- capture the response, harden the retry

igor commented

2026-08-09 15:09:39 +00:00

Collaborator

The security gate failed to produce a parseable verdict THREE consecutive times on igor#480 (2026-08-09 ~01:05, 04:19, 05:53) -- "no verdict from the reviewer after 2 attempts" -- while every other model surface was healthy all night (.health clean throughout). The failure is deterministic and content-shaped, not transient: #480's diff rewrites lib/automerge.sh's header COMMENT describing merge-gate security semantics, and the security reviewer plausibly responds to gate-describing prose with something other than the required verdict format.

Deliverables

Diagnose from evidence, not the theory above. The gate's raw model responses are the ground truth: capture/inspect what the reviewer actually returned on a re-run of the #480 diff (preserve the stream/envelope on no-verdict, the way crashlog preserves worktrees -- if that capture doesn't exist, adding it IS part of this ticket; a no-verdict with no artifact is undebuggable, which is exactly what happened here).
Make no-verdict handling smarter than fail-and-block: a third attempt at escalated effort with a sharpened "your ENTIRE response must be the verdict format" reinforcement is the minimum; whatever the captured responses show should drive the real fix (e.g., if the model is writing an essay about the diff's security implications, the prompt needs a harder format contract).
Keep it fail-closed. No-verdict must still block the push -- the fix is making the verdict arrive, never assuming safety on silence.
Tests: extend the security-gate test coverage with the parse/retry path (a stub returning non-verdict text twice then a valid verdict -> proceeds; three non-verdicts -> blocks with the artifact preserved).

Out of scope

Re-queueing igor#480 itself (the operator is hand-applying that one-paragraph docs change).
Any change to what the gate reviews or its blocking semantics.

Verification

make test green including the new gate tests.
Manual: run the gate against #480's exact diff (preserved in the issue) and confirm either a verdict arrives or the artifact is captured for diagnosis.

The security gate failed to produce a parseable verdict THREE consecutive times on igor#480 (2026-08-09 ~01:05, 04:19, 05:53) -- "no verdict from the reviewer after 2 attempts" -- while every other model surface was healthy all night (`.health` clean throughout). The failure is deterministic and content-shaped, not transient: #480's diff rewrites `lib/automerge.sh`'s header COMMENT describing merge-gate security semantics, and the security reviewer plausibly responds to gate-describing prose with something other than the required verdict format. ## Deliverables 1. **Diagnose from evidence, not the theory above.** The gate's raw model responses are the ground truth: capture/inspect what the reviewer actually returned on a re-run of the #480 diff (preserve the stream/envelope on no-verdict, the way crashlog preserves worktrees -- if that capture doesn't exist, adding it IS part of this ticket; a no-verdict with no artifact is undebuggable, which is exactly what happened here). 2. **Make no-verdict handling smarter than fail-and-block:** a third attempt at escalated effort with a sharpened "your ENTIRE response must be the verdict format" reinforcement is the minimum; whatever the captured responses show should drive the real fix (e.g., if the model is writing an essay about the diff's security implications, the prompt needs a harder format contract). 3. **Keep it fail-closed.** No-verdict must still block the push -- the fix is making the verdict arrive, never assuming safety on silence. 4. **Tests:** extend the security-gate test coverage with the parse/retry path (a stub returning non-verdict text twice then a valid verdict -> proceeds; three non-verdicts -> blocks with the artifact preserved). ## Out of scope - Re-queueing igor#480 itself (the operator is hand-applying that one-paragraph docs change). - Any change to what the gate reviews or its blocking semantics. ## Verification - `make test` green including the new gate tests. - Manual: run the gate against #480's exact diff (preserved in the issue) and confirm either a verdict arrives or the artifact is captured for diagnosis.

igor added the

Agent

label

2026-08-09 15:09:39 +00:00

igor self-assigned this

2026-08-09 15:26:58 +00:00

~~igor referenced this issue 2026-08-09 15:43:37 +00:00~~

issue #491: fix: security gate no-verdict is deterministic on gate-describing diffs -- capture the response, harden the retry #494

igor added spent time

2026-08-09 15:43:38 +00:00

16 minutes 35 seconds

igor referenced this issue from a commit