feat: sitemap/RSS discovery + per-source ledger + candidate promotion #99
No reviewers
Labels
No labels
Agent
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
joshtronic/igor!99
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/reading-sitemap-rss-ledger"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Three coupled changes that replace the heuristic discovery+dedupe
in the reading executor with a deterministic, archive-aware loop.
Discovery: sitemap.xml -> RSS/Atom -> HTML fallback.
Each source's homepage gets probed once for the canonical
post-list endpoint. The result is cached in the per-source
ledger header so subsequent ticks do a single fetch (the
canonical sitemap or RSS URL) instead of re-probing 8 paths.
Sitemaps see the whole archive, not just whatever's on the
homepage today; the old HTML scrape only ever saw the recent
slice.
Per-source ledger at memories/reading/sources/.md:
Posts seen from thatgirljen.com
Source: https://thatgirljen.com
Discovery: sitemap (cached at .../sitemap.xml, last checked DATE)
Index
Replaces the filter_unread heuristic (domain+slug substring
match against log.md + journals) with exact-URL dedupe.
Multi-word titles like "A Confession: I'm an AI-First Coder
Now" that the old heuristic missed are now caught precisely.
log.md stays as the cross-source human-readable record and
as the cross-source dedupe signal -- URLs found in log.md get
flipped to [x] in the source's ledger and never re-picked.
Candidate promotion in agent-reflect-read.sh:
Output schema extends with a
promotionsarray. Thereflection sees the full sources.md (already does, for
dedupe) and can promote weight-0 candidates to weight 1 when
today's read shows substantive engagement. Caps at one
promotion per reflection (gentle pool growth). Igor adjusts
his own pool now; no human-bump needed.
Shared is_nav_url() helper applies the same denylist (assets,
login/signup, legal/about/contact/privacy/terms/tos/imprint,
sitemap/feed/rss, tag/category/archive/page indexes, etc.) to
all three discovery paths -- HTML, sitemap, RSS -- via
ledger_append_urls. extract_links no longer applies the filter
itself; the filter is downstream and uniform.
Aggregator sources (HN) fall through to the HTML path naturally
when sitemap/RSS probes fail; HN does have an RSS feed though
(news.ycombinator.com/rss), and the probe will find and use it
on first discovery.
Bounded intake: nothing in this PR caps sitemap entries by
lastmod, so the first ledger build for a large archive will be
fat. Real-world sitemaps for the current source list (personal
blogs) are small enough that this is fine; revisit if a
1000+post sitemap becomes a problem.
Smoke-tested:
dedup/nav-drop, mark_read, load fresh)
keeps real posts including substring-trap /posts/about-...)
Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com
Three coupled changes that replace the heuristic discovery+dedupe in the reading executor with a deterministic, archive-aware loop. 1. Discovery: sitemap.xml -> RSS/Atom -> HTML fallback. Each source's homepage gets probed once for the canonical post-list endpoint. The result is cached in the per-source ledger header so subsequent ticks do a single fetch (the canonical sitemap or RSS URL) instead of re-probing 8 paths. Sitemaps see the whole archive, not just whatever's on the homepage today; the old HTML scrape only ever saw the recent slice. 2. Per-source ledger at memories/reading/sources/<domain>.md: # Posts seen from thatgirljen.com Source: https://thatgirljen.com Discovery: sitemap (cached at .../sitemap.xml, last checked DATE) ## Index - [x] https://thatgirljen.com/2026/02/02/fledgling/ -- read DATE - [ ] https://thatgirljen.com/2026/03/15/another/ Replaces the filter_unread heuristic (domain+slug substring match against log.md + journals) with exact-URL dedupe. Multi-word titles like "A Confession: I'm an AI-First Coder Now" that the old heuristic missed are now caught precisely. log.md stays as the cross-source human-readable record and as the cross-source dedupe signal -- URLs found in log.md get flipped to [x] in the source's ledger and never re-picked. 3. Candidate promotion in agent-reflect-read.sh: Output schema extends with a `promotions` array. The reflection sees the full sources.md (already does, for dedupe) and can promote weight-0 candidates to weight 1 when today's read shows substantive engagement. Caps at one promotion per reflection (gentle pool growth). Igor adjusts his own pool now; no human-bump needed. Shared is_nav_url() helper applies the same denylist (assets, login/signup, legal/about/contact/privacy/terms/tos/imprint, sitemap/feed/rss, tag/category/archive/page indexes, etc.) to all three discovery paths -- HTML, sitemap, RSS -- via ledger_append_urls. extract_links no longer applies the filter itself; the filter is downstream and uniform. Aggregator sources (HN) fall through to the HTML path naturally when sitemap/RSS probes fail; HN does have an RSS feed though (news.ycombinator.com/rss), and the probe will find and use it on first discovery. Bounded intake: nothing in this PR caps sitemap entries by lastmod, so the first ledger build for a large archive will be fat. Real-world sitemaps for the current source list (personal blogs) are small enough that this is fine; revisit if a 1000+post sitemap becomes a problem. Smoke-tested: - All ledger helpers (init, set/get method, append with dedup/nav-drop, mark_read, load fresh) - Parse functions on sample sitemap.xml, RSS, Atom payloads - is_nav_url against representative URL set (drops nav junk, keeps real posts including substring-trap /posts/about-...) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>