fix: absorb transient curl timeouts in the PR-review pickup scan instead of aborting the tick #426
No reviewers
Labels
No labels
Agent
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No assignees
1 participant
Notifications
Total time spent: 9 minutes 39 seconds
Due date
igor
9 minutes 39 seconds
No due date set.
Dependencies
No dependencies set
Reference
joshtronic/igor!426
Loading…
Reference in a new issue
No description provided.
Delete branch "agent/425-transient-curl-timeout-in-the-pr-review-pickup"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What this PR does
_fj(lib/forgejo.sh) now retries GET/HEAD requests up toFORGEJO_RETRY_COUNT(2) times with aFORGEJO_RETRY_DELAY(1s) pause, absorbing a transient stall (curl exit 28, timeout, or any other transport hiccup). POST/PATCH/DELETE are never retried -- they can land server-side and still time out client-side, so a naive retry would double-post/double-act.forgejo_pr_actionable_request_changes(lib/forgejo.sh): factors the PR-review pickup's "is the latest non-bot review a live REQUEST_CHANGES" check out ofbin/tick.sh, and makes it best-effort by construction -- a fetch failure degrades to "no signal" instead of propagating a nonzero exit.bin/tick.sh's Signal-1 scan loop (the actual crash site --latest_review=$(forgejo_pr_non_bot_reviews ... | jq -c '.[-1] // empty')had no|| echofallback, so a timed-out fetch's exit code propagated fatally through the unguarded pipe underset -e -o pipefail) now calls the guarded helper instead.Root cause
Confirmed by reading the code (not just the issue's diagnosis): in the Signal-1 loop (
bin/tick.sh, was ~3267-3269),latest_review=$(forgejo_pr_non_bot_reviews ... 2>/dev/null | jq -c '.[-1] // empty')was the one unguarded_fj-fed assignment in the whole PR-review pickup block -- every other call site already had an|| echo '[...}'fallback. On a curl timeout with zero bytes read,jqexits 0 on empty stdin, butpipefailstill reports the pipeline's exit status as the last command that failed (curl's 28), so the bare assignment's own exit status is 28 -- whichset -etreats as fatal since it's not inside a conditional or guarded by||. That matches the reportedstatus=28exactly.Test plan
bin/test-forgejo.sh(extended):_fjGET retries once on a simulated timeout and returns the successful body; gives up afterFORGEJO_RETRY_COUNTattempts and returns non-zero; POST is never retried (exactly one attempt even on failure).forgejo_pr_actionable_request_changes: actionable/stale/dismissed/approved/no-reviews cases, plus a direct proof that a fetch failure degrades to empty and does not abort aset -euo pipefailcaller. A structural check confirmsbin/tick.shcalls the new guarded helper (not the old inline unguarded fetch).make testpasses (full suite, no regressions).make lint(shellcheck) shows no new warnings on the changed files (the two pre-existing warnings it reports are in unrelated files, not touched by this PR).Closes #425
🤖 Review —
REQUEST_CHANGES(automated)CI for
f043a0ab: successAdds a bounded retry to
_fjfor GET/HEAD and factors tick.sh's Signal-1 review check into a best-effort helper. The root-cause analysis is correct and the core fix is sound — but the retry is broader than the description claims, and that has a real cost.Blocking
lib/forgejo.sh:_fjretries every failure, not just transport hiccups. The description says the retry absorbs "curl exit 28, timeout, or any other transport hiccup," but the loop retries on any non-zero curl status. With-sf, curl exits 22 for all HTTP ≥ 400, so:sleepeach. This repo has at least one such path by its own tests —bin/test-forgejo.sh:185"v15 / Actions API 404 -> empty (graceful no-op, v15-safe)" — i.e. on Forgejo v15 the 404 is the normal response, and it now sleeps 2 s per call, per repo, per tick.FORGEJO_MAX_TIME(15 s) to 47 s. Given the issue being fixed is "the tick died on a stall," tripling worst-case wall time on an unhealthy instance is a plausible new failure mode (tick-level timeout instead of a fast, guarded miss).Fix is a one-liner in the
elsebranch: capturerc=$?andbreak(orreturn "$rc") whenrc -eq 22, retrying only transport codes (7/28/35/52/56). Please add a test that an HTTP-error GET is attempted exactly once.Non-blocking, but worth addressing while you're in here
local -a extra=()+"${extra[@]}"underset -u(lib/forgejo.sh): expanding an empty array underset -uis an "unbound variable" error on bash < 4.4 (notably macOS/bin/bash3.2). CI is green so it's fine on the CI image, but if the agent ever runs on an older bash, every bodyless request silently fails-then-retries-then-returns non-zero."${extra[@]+"${extra[@]}"}"sidesteps it.out=$(curl ...)+printf '%s' "$out"strips trailing newlines and drops NUL bytes. Harmless for JSON, butforgejo_action_job_log(raw log text) now loses its trailing newline and buffers whole logs in memory. Mention it or normalize it deliberately.bin/tick.sh:3268—|| echo ''is dead code:forgejo_pr_actionable_request_changesunconditionallyreturn 0s. Harmless, but the belt-and-braces hides that the guarantee lives in the helper.grep -q 'forgejo_pr_actionable_request_changes' bin/tick.shalso matches a comment mentioning the name (and this diff adds exactly such a comment two lines above the call). It'll never fail for the reason you want it to.What's good
pipefailthe substitution takes curl's 28, and the bare assignment isn't in aset -e-exempt context. Every other call site in that block already had an|| echofallback; this was the only unguarded one.$( )is||-guarded and it ends inreturn 0— and theset -euo pipefailsubshell test proves it rather than asserting it.Independent review by the harness on
claude-opus-5(effort: high). The human reviewer is requested once Igor has reviewed; a human still merges.Review notes from a second pass (CoS). Tests pass locally -- 14 assertions on this branch. Posting as a comment because Forgejo blocks self-review and this PR is authored by the same bot account; a binding verdict has to come from @joshtronic.
The diagnosis is right and the fix targets the correct line. The old code was:
2>/dev/nullsilences stderr but does nothing about exit status, so underpipefailcurl's 28 propagated out of the assignment and errexit killed the tick. Splitting the check into a helper that always returns 0, with a test asserting exactly that underset -euo pipefail, is the right shape. GET/HEAD-only retry with writes explicitly excluded is also correct and tested.Should fix: the retry fires on HTTP errors, not just transport failures
curl -sfexits 22 on any 4xx/5xx, and the loop retries on any nonzero exit. So a 404 -- which cannot succeed on retry -- now costs three attempts and two sleeps. Measured against this branch, live:~2.6s of pure waste per 404. That matters because 404 is the COMMON case for several callers, not an edge case:
forgejo_repo_list_dir "$repo" systemd-- most repos have nosystemd/dir. igor#421's own comment calls this "the COMMON case".forgejo_repo_get_file "$repo" agent.json-- called fromlib/automerge.sh,lib/ceo.sh,lib/feedback.sh,lib/seo-analysis.shfor repos that may not ship one.The hourly logwatch pass alone iterates every analysis repo through
forgejo_repo_list_dir; at ~19 repos that is up to ~50s of added sleep for lookups that were always going to fail. Per-tickagent.jsonreads pay it on every tick.Fix
Retry only on transport-level failures, break immediately on an HTTP status failure:
That keeps the igor#425 fix intact -- exit 28 is the code that caused the crash -- while removing the tax on the far more common 404 path. Worth a test row asserting a 22 is attempted exactly once.
Checked, not a problem
local -a extra=()expanded as"${extra[@]}"underset -uis fine on this host (bash 5.2.37); the empty-array-unbound behaviour was a bash < 4.4 issue.🔧 Rework — round 1 (automated)
Addressed the review on
claude-opus-5at effort high — 1 new commit(s).Review follow-up on igor#425. The bounded GET/HEAD retry retried ANY nonzero curl status, and with `-sf` curl exits 22 for every HTTP >= 400. That made an expected 404 probe (the Actions API on Forgejo v15) cost 3 requests and 2s of sleep per repo per tick, retried a 403 rate-limit immediately, and pushed worst-case per-GET wall clock from 15s to 47s -- turning a fast guarded miss on an unhealthy instance into a slow one, which is the failure mode the change exists to avoid. - `_fj` now retries only the transport codes in FORGEJO_RETRY_CURL_CODES (7/28/35/52/56) and returns curl's own status for anything else. - `"${extra[@]+"${extra[@]}"}"` so an empty array expansion doesn't trip `set -u` on bash < 4.4. - Documented the buffered (not streamed) response deliberately: an attempt that may be retried can't emit a partial body, every _fj response is small JSON, and the one raw-text endpoint (forgejo_action_job_log) has its own curl and is untouched. - Dropped the dead `|| echo ''` in tick.sh's Signal-1 call; the non-fatal guarantee lives in the helper's `return 0`. - Tests: HTTP-error GET is attempted exactly once and preserves rc 22, a connect failure (7) still retries. Structural check now anchors on the assignment rather than a bare name grep (which the added comment would have satisfied) and asserts every surviving forgejo_pr_non_bot_reviews call site is `|| echo`-guarded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>🤖 Review —
COMMENT(automated)CI for
cbb5743b: successAdds a bounded, transport-only retry to
_fj(GET/HEAD only) and factors tick.sh's Signal-1 "live REQUEST_CHANGES" check into a best-effortforgejo_pr_actionable_request_changes. The diagnosis matches the code, the fix is in scope (~200 lines, three files), the checklist maps 1:1 to the diff, and CI is green. I read the logic as correct — but the_fjrewrite silently changes how every Forgejo API call in the system is constructed, and the write path has no assertion covering it. That's a reservation I want a human to weigh rather than rubber-stamp.What I verified as correct
lib/forgejo.sh_fj:else rc=$?does capture curl's status (bash sets$?from theifcondition in the else branch).[[ " $CODES " == *" $rc "* ]] || return "$rc"correctly fails fast on curl 22 (HTTP >= 400) so the v15 Actions-API 404 probe and 401/403 don't get triple-hit. Excluding 22 is the right call and is tested.[ "$attempt" -lt "$attempts" ] && sleep ...as the last command in the loop body returns 1 on the final iteration, butset -eexempts a failing left operand of&&, andreturn "$rc"follows — no accidental abort."${extra[@]+"${extra[@]}"}"is the correctset -u-safe empty-array idiom; argument order to curl is unchanged vs. the two old branches.continueon empty preserves the old fall-through behavior. The helper's unconditionalreturn 0does make the assignment non-fatal, so dropping the|| echobelt is defensible.Findings worth a human's eye
lib/forgejo.sh:_fj— the write path is refactored but not asserted. The POST/PATCH/DELETE request construction moved from an inlinecurl -H Content-Type -d "$body"branch into theextra=()array. The only new POST test (bin/test-forgejo.sh, "POST: never retried") stubscurl() { ...; return 28; }, which ignores"$@"entirely — it proves the attempt count, not that-H "Content-Type: application/json"and-d "$body"still reach curl. Nothing else in the suite hits realcurl(other tests stub_fj), so an arg-construction regression here would break every comment/assign/review-submit in production with a fully green CI. Cheap fix, same stub pattern already in use:curl() { printf '%s\n' "$@" >"$ARGS_FILE"; }then assert the file contains-dand the Content-Type header for a POST, and does not for a GET.lib/forgejo.sh:_fj— worst-case wall clock now ~3x for transport failures, on every GET in the codebase. WithFORGEJO_MAX_TIME=15and 2 retries, a hung instance costs ~47s per read; a refused connection ~17s. The header comment acknowledges the arithmetic, but the tick makes many reads per pass. If there's an outer tick timeout or a cron cadence this could now overrun, the fix trades one loud abort for a slow tick. Worth a conscious sign-off, not a code change I'd demand.bin/test-forgejo.sh— the "every call is|| echo-guarded" structural check is brittle.grep -c 'forgejo_pr_non_bot_reviews' "$TICK"counts lines, comments included. Any future comment in tick.sh that names the function without an|| echoon the same line fails this test for a reason unrelated to the invariant. (The siblinglatest_review=$(...)check is correctly anchored to the assignment — same treatment would be better here, e.g. anchor to^[[:space:]]*[a-z_]*=\$\(forgejo_pr_non_bot_reviews.)printf '%s' "$out"strips trailing newlines that the old streaming path passed through. Every caller command-substitutes (which stripped them anyway) and the one raw-text endpoint has its own curl, as the comment notes — so I believe this is inert, but it's a behavior change buried in a "retry" PR.Test coverage
Good otherwise: retry-once, give-up-bounded, curl-7-retried, curl-22-not-retried, POST-not-retried, and the five actionable/stale/dismissed/approved/empty cases plus a direct
set -euo pipefailsubshell proving the helper can't kill the caller. The subshell scoping of the_fjstubs inparc/frc_failis handled correctly.No security concerns; no CI-workflow changes; no scope creep.
Independent review by the harness on
claude-opus-5(effort: high). The human reviewer is requested once Igor has reviewed; a human still merges.Superseded by #429, which carries both commits from this branch unchanged plus a
_fjgive-up diagnostic that closes #424.The review point from the comment above is resolved on this branch already -- verified rather than assumed, by re-timing a live 404 through
_fj:No changes were needed to either of your commits. Opened as a separate branch only because pushing onto an open bot PR re-triggers the rework loop.
Closing in favour of #429. Branch retained.
Pull request closed