fix: scope _fj retry to transport failures + log when it gives up #429
No reviewers
Labels
No labels
Agent
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No assignees
2 participants
Notifications
Total time spent: 13 minutes 11 seconds
Due date
igor
13 minutes 11 seconds
No due date set.
Dependencies
No dependencies set
Reference
joshtronic/igor!429
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/425-scope-retry-to-transport"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What this PR does
Supersedes #426 -- carries both of its commits unchanged, plus one addition that closes #424. Opened as a separate branch rather than pushed onto the bot's, which collides with the rework loop.
_fj, never HTTP errors (from #426)_fjexhausts its retries (new -- closes #424)Review of the inherited work (verified, not assumed)
Both #426 commits check out. I re-ran the suites and re-measured the thing I flagged in review:
That was 2.89s before the transport-code scoping -- so the 404 tax is gone, and 404 is the common path for
forgejo_repo_list_dir/forgejo_repo_get_fileacross automerge, ceo, feedback and seo-analysis. No changes needed to either commit.The addition
igor#424 reported a tick dying
status=28with no error line at all. #425's work stops that path aborting a tick, but an exhausted_fjstill returned nonzero silently, so the next occurrence would be just as opaque. Now it logs the method, path, attempt count and curl exit code.Kept deliberately narrow:
agent.jsonis the common path and logging it would bury the signal._fjcaller command-substitutes the response and several guard with|| truethen use the captured value -- on stdout this diagnostic would silently become that value. There is an explicit test for it.Test plan
bash bin/test-forgejo.shpasses, including 5 new assertionsmake testpasses (24 suites)shellcheck lib/forgejo.sh bin/test-forgejo.shclean[agent] forgejo: GET /repos/acme/x/pulls/1/reviews failed after 3 attempt(s) (curl exit 28)One thing worth knowing about the suite
My first draft of these assertions used a
hashelper.bin/test-forgejo.shdefines onlyeq-- so three assertions died ashas: command not foundon stderr while the suite still printed "all checks passed", because it runs underset -uo pipefailwithout-eandFAILwas never incremented. Caught by reading the output rather than the exit code; rewritten in this file'seq+grepidiom.That is a latent trap for anything else added to this suite -- a typo'd helper name is a silently skipped test, not a failure. Not fixing it here (it would touch every
bin/test-*.sh), but it is worth a follow-up.Closes #424
Closes #425
Review follow-up on igor#425. The bounded GET/HEAD retry retried ANY nonzero curl status, and with `-sf` curl exits 22 for every HTTP >= 400. That made an expected 404 probe (the Actions API on Forgejo v15) cost 3 requests and 2s of sleep per repo per tick, retried a 403 rate-limit immediately, and pushed worst-case per-GET wall clock from 15s to 47s -- turning a fast guarded miss on an unhealthy instance into a slow one, which is the failure mode the change exists to avoid. - `_fj` now retries only the transport codes in FORGEJO_RETRY_CURL_CODES (7/28/35/52/56) and returns curl's own status for anything else. - `"${extra[@]+"${extra[@]}"}"` so an empty array expansion doesn't trip `set -u` on bash < 4.4. - Documented the buffered (not streamed) response deliberately: an attempt that may be retried can't emit a partial body, every _fj response is small JSON, and the one raw-text endpoint (forgejo_action_job_log) has its own curl and is untouched. - Dropped the dead `|| echo ''` in tick.sh's Signal-1 call; the non-fatal guarantee lives in the helper's `return 0`. - Tests: HTTP-error GET is attempted exactly once and preserves rc 22, a connect failure (7) still retries. Structural check now anchors on the assignment rather than a bare name grep (which the added comment would have satisfied) and asserts every surviving forgejo_pr_non_bot_reviews call site is `|| echo`-guarded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>🤖 Review —
REQUEST_CHANGES(automated)CI for
ab061669: successAdds a bounded, transport-only retry to
_fj, extracts the PR-review Signal-1 check into a best-effort helper, and logs when_fjexhausts its retries. Well-tested and honestly described — but the new give-up diagnostic is swallowed on the exact code path that motivated igor#424, which I think needs fixing before merge.Blocking
1. The igor#424 diagnostic never reaches the path igor#424 was filed about. (
bin/tick.sh:3272,lib/forgejo.shhelper)The description's rationale is: "#425's work stops that path aborting a tick, but an exhausted
_fjstill returned nonzero silently, so the next occurrence would be just as opaque." But stderr is discarded twice on that very path:lib/forgejo.sh:reviews=$(forgejo_pr_non_bot_reviews "$repo" "$number" "$bot" 2>/dev/null) || reviews='[]'bin/tick.sh:3272:latest_review=$(forgejo_pr_actionable_request_changes "$repo_full" "$pr_num" "$BOT_USER" 2>/dev/null)So the next exit-28 in the PR-review pickup scan — the reported occurrence, per your own test comment ("two exit-28 ticks landing in this PR-review pickup scan") — will still produce zero output. It won't kill the tick anymore (that's #425), but the #424 half of this PR is a no-op precisely where it was asked for. Both of those
2>/dev/nulls are added by this diff, so it's in scope to fix.Fixed looks like: drop the
2>/dev/nullon the tick.sh call (the helper'sreturn 0already provides the non-fatal guarantee, per your own comment there), and narrow or drop the inner one so the_fjgive-up line survives while jq noise stays suppressed. Add an assertion that the give-up line is visible throughforgejo_pr_actionable_request_changes, not just through_fjdirectly — the current tests only exercise_fjin isolation, which is why this gap passes CI.Non-blocking, but please answer
2.
printf '%s' "$out"drops the trailing newline that curl used to emit. (lib/forgejo.sh,_fj)The comment says the cost is "a trailing newline that
$( )would have stripped regardless" — true for command substitution, but not for a caller that pipes_fjoutput._fj GET ... | while IFS= read -r lineloses the final (often only) line, sincereadat EOF without a delimiter returns non-zero.printf '%s\n' "$out"restores the old byte stream exactly at zero cost. Unless you've grepped every_fjcall site for pipe-into-read(I can't from the diff), I'd just use'%s\n'.3. Worst-case wall clock on a wedged instance. A hung
GETis now3 × FORGEJO_MAX_TIME + 2 × FORGEJO_RETRY_DELAY≈ 47s per call, and the pickup scan does this per PR per repo. The transport-code scoping keeps the common 404 path fast (good — and the 2.89s → 0.30s measurement is the right thing to have checked), but a genuinely unreachable host now costs 3× per read across the whole scan. Worth a sentence on why that's acceptable, or a global deadline.4. Fragile structural assertion.
bin/test-forgejo.sh:grep -c 'forgejo_pr_non_bot_reviews' "$TICK"vsgrep -c '|| echo'on the same lines will false-fail if a future call site puts|| echoon a continuation line, and passes vacuously if the count is 0 (which it may be now — this diff removed what looks like the only tick.sh call site). Not worth blocking on; just be aware it may currently be asserting0 == 0.Verified
case "$method" in GET|HEAD)gates attempts, and non-transport codesreturn "$rc"before any sleep, so POST/PATCH/DELETE can't double-act and a 22 isn't retried.local -a extra=()with"${extra[@]+"${extra[@]}"}"is the rightset -u-safe idiom.forgejo_pr_actionable_request_changesis behaviour-preserving vs. the inlined block it replaces (same REQUEST_CHANGES / !stale / !dismissed predicate, same last-match-wins loop semantics) and genuinely ends inreturn 0.set -euo pipefailsubshell test is the right shape for the regression being guarded.log()fallback guard (declare -F) is safe and matches the statedlib/http-reap.shprecedent.The note about
bin/test-*.shsilently passing on a typo'd helper name is a good catch and correctly left out of this diff — please do file it.Independent review by the harness on
claude-opus-5(effort: high). The human reviewer is requested once Igor has reviewed; a human still merges.🔧 Rework — round 1 (automated)
Addressed the review on
claude-opus-5at effort high — 5 new commit(s).🤖 Review —
COMMENT(automated)CI for
94c7b953: successScopes
_fj's new bounded retry to transport-only curl codes and adds a stderr give-up log; carries #426's two commits plus one new one. CI green, tests are unusually thorough, checklist matches the diff. Nothing I can prove is broken — but a few reservations I'd want you to weigh before merging, so COMMENT rather than approve.Findings
lib/forgejo.shforgejo_pr_non_bot_reviews— the new docstring makes a claim the diff only half-delivers. The comment says jq's stderr is suppressed here "rather than by the callers (igor#424): both of them want the parse noise gone… blanketing the whole function with2>/dev/nullalso swallows_fj's give-up line." Only one call site is touched in the diff (bin/tick.sh:3335). If the otherforgejo_pr_non_bot_reviewscall site in tick.sh still carries a2>/dev/null, the give-up line is still swallowed there and the stated rationale doesn't hold for it. The new structural test only asserts|| echoon those lines, not the absence of2>/dev/null— so nothing catches it. Please confirm the second call site (and add the stderr assertion if it applies)._fjnow normalizes the response's trailing bytes for every caller.out=$(curl …)strips all trailing newlines andprintf '%s\n' "$out"re-emits exactly one. Callers that command-substitute or pipe tojqcan't tell (and the pipe-into-readcase is tested), but any caller that writes_fjoutput straight to a file or compares bytes now seesfoo\n\n\n→foo\n, and a body with no trailing newline gains one. The other call sites aren't in the diff, so I can't verify none of them do that.Retry amplification on legitimately slow reads, not just blips. A large response that genuinely exceeds
--max-time 15(job-log / CI-log fetches go through_fjGET) exits 28 — indistinguishable from a transient stall — so it now burns 3 × 15s + 2 × 1s ≈ 47s and still fails, where it used to fail in 15s. The description accepts the 47s worst case for an unreachable instance; the "response is just big" case is the same cost for a deterministic failure. Worth confirming the log-fetch path can't hit it. Related: the full body is now buffered in a shell variable rather than streamed, which matters most on exactly that path.Process, not code: this PR closes #424 and #425 and supersedes #426. #426 won't auto-close — remember to close it manually so the rework loop doesn't keep working the stale branch.
Test-coverage notes
_fjthrough the realforgejo_pr_actionable_request_changes(stubbing onlycurl/sleep) is the right shape — that's the assertion that actually protects igor#424's reported path.grep 'forgejo_pr_non_bot_reviews' "$TICK" | grep -c '|| echo'matches lines, including comments. A future comment mentioning the function name, or a call whose|| echowraps to the next line, fails this suite for the wrong reason. Consider anchoring it to the assignment shape the way thelatest_review=check already is.has: command not foundtrap you documented (missing helper ⇒ silently skipped test, suite still prints "all checks passed") is a genuinely important find. Agreed it's out of scope here, but it should get an issue — until then, everybin/test-*.shcan green-light assertions that never ran.No security concerns; no CI-config changes; diff is in scope and under budget.
Independent review by the harness on
claude-opus-5(effort: high). The human reviewer is requested once Igor has reviewed; a human still merges.