fix: agent-read resilience -- ARG_MAX, error context, retry, fragments #81

Merged

joshtronic merged 1 commit from fix/agent-read-resilience into master

2026-05-22 04:32:36 +00:00

joshtronic commented

2026-05-22 04:32:02 +00:00

Owner

A single reading tick can blow up four different ways and end the
whole tick. Each path now either survives the failure or gives a
useful error.

ARG_MAX: the real fire.
agent-read.sh passed the truncated HTML (up to 200KB) to jq as
--arg u "$USER_MESSAGE". The full payload landed on the
process's argv, hit the kernel's ARG_MAX, and exec'd jq with
"Argument list too long". The tick died there.

Fix: build the request payload via --rawfile against a temp
file. jq reads the file content as a literal string and embeds
it correctly in the JSON. No argv pressure. Verified with a
200KB synthetic payload -- jq reports the full content length
preserved.
Vague error context.
The old "API call failed for X" swallowed the curl exit code
and HTTP status under 2>/dev/null. Could be anything: rate
limit, auth failure, DNS, 5xx, the ARG_MAX above.

Fix: split curl exit (network/transport) from HTTP status
(application). Drop -f, use -w '%{http_code}' -o file, and
read the error body via jq. Errors now say "curl failed (exit
22): " or "API returned HTTP 529 -- Overloaded".
No retry across URLs/sources.
discretionary-read.sh picked ONE URL via try_source -> shuf -n 1
and bailed on first agent-read failure, even when the source's
homepage had 20 other fresh URLs sitting right there.

Fix: factored try_source into list_source_candidates (returns
ALL fresh URLs from a source). Restructured the main loop into
an attempt-bounded retry. Up to MAX_ATTEMPTS=3 agent-read calls
per tick. Failed URLs drop from the in-memory pool for this
run only -- still re-tryable on future ticks since the failure
was probably transient. When the current source's candidates
are exhausted, sample a new source.
URL fragments leaking through.
joshtronic.com linked to https://www.11ty.dev/#why-should-you-
use-eleventy. The fragment made it look distinct from the
11ty.dev homepage and the picker happily handed agent-read a
200KB marketing page -- which triggered #1.

Fix: extract_links strips #... fragments and drops bare-
domain URLs (https://example.com, https://example.com/) since
those are usually the source's own homepage or a footer link
to an aggregator, not a content page.

Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com

A single reading tick can blow up four different ways and end the whole tick. Each path now either survives the failure or gives a useful error. 1. ARG_MAX: the real fire. agent-read.sh passed the truncated HTML (up to 200KB) to jq as `--arg u "$USER_MESSAGE"`. The full payload landed on the process's argv, hit the kernel's ARG_MAX, and exec'd jq with "Argument list too long". The tick died there. Fix: build the request payload via `--rawfile` against a temp file. jq reads the file content as a literal string and embeds it correctly in the JSON. No argv pressure. Verified with a 200KB synthetic payload -- jq reports the full content length preserved. 2. Vague error context. The old "API call failed for X" swallowed the curl exit code and HTTP status under `2>/dev/null`. Could be anything: rate limit, auth failure, DNS, 5xx, the ARG_MAX above. Fix: split curl exit (network/transport) from HTTP status (application). Drop `-f`, use `-w '%{http_code}' -o file`, and read the error body via jq. Errors now say "curl failed (exit 22): <stderr line>" or "API returned HTTP 529 -- Overloaded". 3. No retry across URLs/sources. discretionary-read.sh picked ONE URL via try_source -> shuf -n 1 and bailed on first agent-read failure, even when the source's homepage had 20 other fresh URLs sitting right there. Fix: factored try_source into list_source_candidates (returns ALL fresh URLs from a source). Restructured the main loop into an attempt-bounded retry. Up to MAX_ATTEMPTS=3 agent-read calls per tick. Failed URLs drop from the in-memory pool for this run only -- still re-tryable on future ticks since the failure was probably transient. When the current source's candidates are exhausted, sample a new source. 4. URL fragments leaking through. joshtronic.com linked to https://www.11ty.dev/#why-should-you- use-eleventy. The fragment made it look distinct from the 11ty.dev homepage and the picker happily handed agent-read a 200KB marketing page -- which triggered #1. Fix: extract_links strips `#...` fragments and drops bare- domain URLs (https://example.com, https://example.com/) since those are usually the source's own homepage or a footer link to an aggregator, not a content page. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

joshtronic added 1 commit

2026-05-22 04:32:02 +00:00

fix: agent-read resilience -- ARG_MAX, error context, retry, fragments

Lint / check-sync (push) Successful in 6s

Details

Lint / check-sync (pull_request) Successful in 4s

Details

5abd70dc09

A single reading tick can blow up four different ways and end the
whole tick. Each path now either survives the failure or gives a
useful error.

1. ARG_MAX: the real fire.
   agent-read.sh passed the truncated HTML (up to 200KB) to jq as
   `--arg u "$USER_MESSAGE"`. The full payload landed on the
   process's argv, hit the kernel's ARG_MAX, and exec'd jq with
   "Argument list too long". The tick died there.

   Fix: build the request payload via `--rawfile` against a temp
   file. jq reads the file content as a literal string and embeds
   it correctly in the JSON. No argv pressure. Verified with a
   200KB synthetic payload -- jq reports the full content length
   preserved.

2. Vague error context.
   The old "API call failed for X" swallowed the curl exit code
   and HTTP status under `2>/dev/null`. Could be anything: rate
   limit, auth failure, DNS, 5xx, the ARG_MAX above.

   Fix: split curl exit (network/transport) from HTTP status
   (application). Drop `-f`, use `-w '%{http_code}' -o file`, and
   read the error body via jq. Errors now say "curl failed (exit
   22): <stderr line>" or "API returned HTTP 529 -- Overloaded".

3. No retry across URLs/sources.
   discretionary-read.sh picked ONE URL via try_source -> shuf -n 1
   and bailed on first agent-read failure, even when the source's
   homepage had 20 other fresh URLs sitting right there.

   Fix: factored try_source into list_source_candidates (returns
   ALL fresh URLs from a source). Restructured the main loop into
   an attempt-bounded retry. Up to MAX_ATTEMPTS=3 agent-read calls
   per tick. Failed URLs drop from the in-memory pool for this
   run only -- still re-tryable on future ticks since the failure
   was probably transient. When the current source's candidates
   are exhausted, sample a new source.

4. URL fragments leaking through.
   joshtronic.com linked to https://www.11ty.dev/#why-should-you-
   use-eleventy. The fragment made it look distinct from the
   11ty.dev homepage and the picker happily handed agent-read a
   200KB marketing page -- which triggered #1.

   Fix: extract_links strips `#...` fragments and drops bare-
   domain URLs (https://example.com, https://example.com/) since
   those are usually the source's own homepage or a footer link
   to an aggregator, not a content page.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>