fix: gate the shadow review to validated repos (don't review not-ready repos) #244

New issue

Closed

opened 2026-06-25 03:40:29 +00:00 by igor · 0 comments

igor commented

2026-06-25 03:40:29 +00:00

Collaborator

Problem

The shadow review (do_review_tick) runs on the analysis set (every
bot-accessible repo), so it reviews repos that aren't validated -- including
repos with an open "not ready for the agent" onboarding ticket.

Live example: it posted a REQUEST_CHANGES verdict and requested the human on
joshtronic/devopsafterdark.com#3, a repo whose onboarding ticket #1 ("Repo
not ready for the agent") is still open. #243 correctly prevented the autonomous
rework there (it handed to the human with a "not validated" comment) -- but
the review itself shouldn't have run at all.

A verdict on an unvalidated repo can never become a merge signal (an unvalidated
repo can't auto-merge), so reviewing it collects no useful auto-merge data --
it's just bot footprint on a not-ready repo.

Fix

Gate do_review_tick's repo selection to the validated set, mirroring #243's
rework gate. The function's own header comment already says "validated work set"
-- this aligns the code with that stated intent.

In the repo loop (bin/tick.sh ~L2495-2511, iterating ANALYSIS_REPOS_JSON),
right after the repo= line, skip unvalidated repos:

repo=$(jq -r '.full_name' <<<"$repo_line")
maintenance_repo_validated "$repo" || continue

maintenance_repo_validated already exists (checks VALIDATED_REPOS_JSON
membership; it's exactly what #243 used). It naturally also skips onboarding
repos, since an open onboarding ticket keeps a repo out of the validated set.

Watch out for

Gate ONLY the review's repo selection. Don't touch the #243 rework gate.
The harness's OWN PRs must still be reviewed -- igor IS validated (it has
tests + CI), so it stays in scope. Don't regress that.
Update the CLAUDE.md shadow-review bullet: it currently says "Runs on the
ANALYSIS set, not the validated work set: a verdict is information, useful on
every repo." Flip it to the validated set and drop the "useful on every repo"
rationale -- we've decided the opposite: an unvalidated verdict can't graduate,
so it's just footprint.
bash -n bin/tick.sh clean; bash bin/check-sync.sh passes.

Definition of done

do_review_tick skips repos not in the validated set -- no verdict comment and
no review-request on unvalidated / onboarding repos.
Validated repos (including igor itself) still get reviewed, unchanged.
Regression intent: a PR on an unvalidated repo (stonks-like / devopsafterdark-
like) gets NO shadow-review verdict at all.
CLAUDE.md shadow-review bullet updated to say validated-set.
bash -n bin/tick.sh clean; check-sync passes.

## Problem The shadow review (`do_review_tick`) runs on the **analysis set** (every bot-accessible repo), so it reviews repos that aren't validated -- including repos with an open "not ready for the agent" onboarding ticket. Live example: it posted a `REQUEST_CHANGES` verdict and requested the human on `joshtronic/devopsafterdark.com#3`, a repo whose onboarding ticket #1 ("Repo not ready for the agent") is still open. #243 correctly prevented the autonomous **rework** there (it handed to the human with a "not validated" comment) -- but the review itself shouldn't have run at all. A verdict on an unvalidated repo can never become a merge signal (an unvalidated repo can't auto-merge), so reviewing it collects no useful auto-merge data -- it's just bot footprint on a not-ready repo. ## Fix Gate `do_review_tick`'s repo selection to the validated set, mirroring #243's rework gate. The function's own header comment already says "validated work set" -- this aligns the code with that stated intent. In the repo loop (`bin/tick.sh` ~L2495-2511, iterating `ANALYSIS_REPOS_JSON`), right after the `repo=` line, skip unvalidated repos: ```bash repo=$(jq -r '.full_name' <<<"$repo_line") maintenance_repo_validated "$repo" || continue ``` `maintenance_repo_validated` already exists (checks `VALIDATED_REPOS_JSON` membership; it's exactly what #243 used). It naturally also skips onboarding repos, since an open onboarding ticket keeps a repo out of the validated set. ## Watch out for - Gate ONLY the review's repo selection. Don't touch the #243 rework gate. - The harness's OWN PRs must still be reviewed -- `igor` IS validated (it has tests + CI), so it stays in scope. Don't regress that. - Update the CLAUDE.md shadow-review bullet: it currently says *"Runs on the ANALYSIS set, not the validated work set: a verdict is information, useful on every repo."* Flip it to the validated set and drop the "useful on every repo" rationale -- we've decided the opposite: an unvalidated verdict can't graduate, so it's just footprint. - `bash -n bin/tick.sh` clean; `bash bin/check-sync.sh` passes. ## Definition of done - `do_review_tick` skips repos not in the validated set -- no verdict comment and no review-request on unvalidated / onboarding repos. - Validated repos (including `igor` itself) still get reviewed, unchanged. - Regression intent: a PR on an unvalidated repo (stonks-like / devopsafterdark- like) gets NO shadow-review verdict at all. - CLAUDE.md shadow-review bullet updated to say validated-set. - `bash -n bin/tick.sh` clean; `check-sync` passes.

igor self-assigned this

2026-06-25 03:43:01 +00:00

igor referenced this issue from a pull request that will close it,

2026-06-25 03:44:12 +00:00

refactor: restrict PR review tick to validated repos only #245

igor added spent time

2026-06-25 03:44:12 +00:00

51 seconds

igor removed their assignment

2026-06-25 03:44:13 +00:00

igor referenced this issue from a commit

2026-06-25 03:45:06 +00:00