test: pin the review directive's VERDICT/===BODY=== contract to review_parse_response #457

New issue

Closed

opened 2026-07-30 17:30:23 +00:00 by igor · 0 comments

igor commented

2026-07-30 17:30:23 +00:00

Collaborator

Split out of #455, where the shadow review flagged it as an unrelated
drive-by. It was: I had bin/lib/review-directive.md open for a one-line
change and the test rode along.

What is uncovered

bin/lib/review-directive.md tells the model to emit VERDICT: <token> above
a ===BODY=== sentinel. review_parse_response in bin/tick.sh accepts
exactly three tokens and that one sentinel. Nothing pins the two together.

Edit the directive's output-format block without touching the parser and every
review becomes "no parseable verdict after 2 attempts", which retries twice per
tick, forever, and reads like a model failure rather than a text mismatch. The
review pass would be silently dead across the fleet and the journal would blame
the model.

What the test should do

Assert the handshake only -- the rubric is prose and no assertion can tell you
whether it produces good verdicts. Two properties worth pinning:

every verdict token the directive advertises round-trips through the parser
the sentinel the directive names is the one the parser splits on

Lift the real parser out of bin/tick.sh with sed rather than reimplementing
it. A hand-rolled copy would happily agree with a directive that the shipping
parser rejects, which is the one failure this is meant to catch.

review_parse_response ends in jq -n, so the suite has to guard on jq like
its siblings do, or it reports a directive/parser mismatch that is not there on
a host without jq. That guard is the skip-safety contract bin/check-sync.sh
expects of every bin/test-*.sh.

A working version exists on the feat/rework-adjudication branch (added in
e388efc, fixed in e68a83c, removed when #455 was narrowed) if recovering it
beats rewriting it.

Not in scope

The directive's rubric wording. That is #456, still awaiting a decision.

Split out of #455, where the shadow review flagged it as an unrelated drive-by. It was: I had `bin/lib/review-directive.md` open for a one-line change and the test rode along. ## What is uncovered `bin/lib/review-directive.md` tells the model to emit `VERDICT: <token>` above a `===BODY===` sentinel. `review_parse_response` in `bin/tick.sh` accepts exactly three tokens and that one sentinel. Nothing pins the two together. Edit the directive's output-format block without touching the parser and every review becomes "no parseable verdict after 2 attempts", which retries twice per tick, forever, and reads like a model failure rather than a text mismatch. The review pass would be silently dead across the fleet and the journal would blame the model. ## What the test should do Assert the handshake only -- the rubric is prose and no assertion can tell you whether it produces good verdicts. Two properties worth pinning: - every verdict token the directive advertises round-trips through the parser - the sentinel the directive names is the one the parser splits on Lift the real parser out of `bin/tick.sh` with `sed` rather than reimplementing it. A hand-rolled copy would happily agree with a directive that the shipping parser rejects, which is the one failure this is meant to catch. `review_parse_response` ends in `jq -n`, so the suite has to guard on `jq` like its siblings do, or it reports a directive/parser mismatch that is not there on a host without `jq`. That guard is the skip-safety contract `bin/check-sync.sh` expects of every `bin/test-*.sh`. A working version exists on the `feat/rework-adjudication` branch (added in e388efc, fixed in e68a83c, removed when #455 was narrowed) if recovering it beats rewriting it. ## Not in scope The directive's rubric wording. That is #456, still awaiting a decision.

igor self-assigned this

2026-07-30 17:59:43 +00:00

igor referenced this issue from a pull request that will close it,

2026-07-30 18:03:59 +00:00

test: pin the review directive's VERDICT/===BODY=== contract to review_parse_response #458

igor added spent time

2026-07-30 18:04:00 +00:00

3 minutes 55 seconds

igor removed their assignment

2026-07-30 18:04:00 +00:00

igor referenced this issue

2026-07-30 18:20:04 +00:00