Architecture

A reference for contributors. Covers process layout, request lifecycle, and how the major subsystems hand data off.

Process layout

caddy ─┬─> ember (Go binary)
       │     ├─ HTTP API + Fever shim + SPA serve
       │     ├─ Background poller (per-feed adaptive ticker)
       │     ├─ Summary worker pool (Ollama HTTP client)
       │     ├─ DB maintenance goroutine (retention prune / backup / cleanup / OPML / hourly)
       │     ├─ Update-check goroutine (daily GitHub-releases poll; release builds only)
       │     └─ Cluster backfill goroutine (one-time at startup; idempotent)
       │
       └─> SPA (embedded static files served by ember)

ollama  (sidecar) — model storage + inference

The single ember binary embeds the Svelte SPA via embed.FS and serves it under /. The API is /api/*; mobile clients use /fever. The poller, summary worker, and maintenance goroutine all share the database via internal/store.

Package map

cmd/ember/                    main + probe subcommand + DB maintenance + digest sender
internal/api/                 chi router, handlers, middleware (CSRF, rate limit, auth context)
internal/auth/                argon2id passwords, securecookie sessions, WebAuthn (passkeys), RequireAuth/Admin middleware
internal/config/              env-var loading (typed Config)
internal/db/                  SQLite open (write handle + read-only pool), pragmas, embedded migrations (goose)
internal/digest/              SMTP daily-digest builder + sender (multipart/alt + STARTTLS)
internal/feed/                gofeed wrapper + readability fallback fetcher + Discover / DiscoverAll (homepage → one or many feed URLs) + URL normalize (schemeless → https) + CanonicalURL / ClusterID / TitleFingerprint (cross-feed dedup keys)
internal/filters/             matcher (field/op/value), apply outcome combiner
internal/models/              data types shared across packages
internal/opml/                OPML import + export + discovery → subscribe
internal/ttrss/               Tiny Tiny RSS migration — XML export parser (articles → one non-polling "Imported" feed) + live JSON API client (subscriptions + categories→folders, plus starred/archived articles)
internal/poller/              adaptive scheduler, fetch dispatch, summary queue
internal/store/               SQLite CRUD, FTS5 search, app_settings KV, dbops, passkeys, digests, cluster backfill + sibling lookup
internal/summarize/           Summarizer interface + Ollama implementation + noop for tests
internal/sysinfo/             host-detection (RAM/CPU/GPU) + model recommendation
internal/updatecheck/         daily GitHub-releases poll + semver compare; caches the latest tag for the admin-only /api/me hint
internal/urlcheck/            SSRF block (scheme allowlist + private-IP + service-port refusal)
internal/web/                 embed.FS handler for the SPA
web/                          Svelte 5 (runes) source; built via Vite, copied to internal/web/dist

Database schema

SQLite. Migrations in internal/db/migrations/*.sql, applied at startup. Key tables:

• users — argon2id-hashed passwords; admin flag. last_login_at / prev_login_at track logins so the All-Unread window can extend back to the previous visit.
• sessions — server-side rows backing cookies; pruned periodically.
• login_failures — consecutive failed password logins per submitted username, driving the per-account login backoff. Keyed on the username as typed (whether or not the account exists, so the throttle can't be read as an enumeration oracle); cleared on the next successful login for that name and swept hourly once a row is older than 24h.
• feeds — shared across users; URL-unique. Tracks etag/last-modified/error counters.
• subscriptions — (user_id, feed_id) with category, title override, muted flag, summarize flag (per-user AI-summary opt-out, default 1), and position for drag-reorder.
• categories — user-scoped folders with color + position.
• articles — shared across users; per-feed dedup by guid and content_hash. Carries cleaned_html (AI ad-stripped). Also stores canonical_url, cluster_id, and title_fingerprint for cross-feed dedup (see Cross-feed dedup); partial indexes idx_articles_cluster and idx_articles_fp_pub skip empty values so unfilled rows never falsely match.
• article_state — per-user read/star/later flags.
• articles_fts — FTS5 virtual table on title/text/author; kept in sync via triggers. Search is bounded by the admin search window (default 48h, capped at the 1-week retention window).
• boards + board_articles — user-curated collections.
• filters — rule store; engine in internal/filters.
• shares — user-to-user article shares.
• app_settings — global KV (active model, schedules, branding, tuning).
• saved_searches — persisted FTS queries.
• article_tags — per-user tags on individual articles.
• user_digests — per-user opt-in daily-digest config (view, hour/minute UTC, last-sent timestamp).
• passkeys — WebAuthn credentials (credential_id, public_key, sign_count, name, timestamps).
• webauthn_sessions — short-lived ceremony state for in-flight register/login flows; reaped after 5 min.

WAL mode, 256 MiB mmap, busy_timeout=5s. Two handles over the same file:

• Write handle (db.Open) — MaxOpenConns=1, 64 MiB page cache. SQLite has a single writer, and multiple writing connections hit SQLITE_BUSY in a form busy_timeout doesn't cover (BUSY_SNAPSHOT), so the cap is deliberate. It also runs the migrations.
• Read handle (db.OpenRead) — MaxOpenConns=4, opened query_only, 16 MiB page cache per connection. WAL serves readers concurrently with the writer, so the heavy list/count/search queries (Store.reader()) no longer queue behind whatever the poller is writing: worst-case read latency during a fetch drops from ~166ms to ~23ms. query_only means a store method routed here by mistake fails loudly instead of silently reintroducing write contention. It deliberately does not migrate, so it can't race the writer's schema work.

cache_size is per connection, so the read pool uses 16 MiB rather than inheriting the writer's 64 MiB — that keeps the total page-cache budget at roughly 64 + 4×16 MiB instead of 5×64. If OpenRead fails the store falls back to the write handle and behaves exactly as before.

Request lifecycle

HTTPS                Caddy
  └─> :443  ──tls──> ember:8080
                       │
                       ▼
                  chi.Router
                       │
        ┌──────────────┼──────────────┐
        ▼              ▼              ▼
   /api/auth/*    /api/*         /fever  (Fever shim)
                       │              │
                  CSRFVerify          │
                  RequireAuth         │
                  (RequireAdmin)      │
                       │              │
                       ▼              ▼
                   handlers       fever_handlers
                       │              │
                       └─> store ─────┘
                              │
                              ▼
                          SQLite (WAL)

Browsers store a server-side session cookie + a CSRF cookie. State-changing routes (POST/PATCH/DELETE) must echo the CSRF cookie in X-Ember-CSRF.

/api/auth/* carries two independent throttles, because they answer different questions. The per-IP token bucket bounds one source; the per-username backoff in internal/auth/throttle.go bounds one account, which is what a credential-stuffing run spread across many addresses actually attacks. The username throttle is consulted before the user lookup and before any argon2 work, so rejected traffic costs a single indexed read rather than a 64 MiB key derivation.

Poller state machine

Each feed has next_fetch (unix seconds) and an error_count. The poller:

1. Every EMBER_POLL_TICK (default 60s): Tick(ctx) queries FeedsDue (next_fetch ≤ now).
2. Due feeds fan out across EMBER_POLL_CONCURRENCY worker goroutines.
3. Each worker calls Fetcher.Fetch(url, etag, last_modified):
   • 304 → bookkeep last_fetched/next_fetch, error_count = 0.
   • 2xx → parse with gofeed, optionally enrich short bodies via go-readability, upsert articles (the upsert stamps canonical_url, cluster_id, and title_fingerprint so cross-feed dedup keys are populated at ingest, not only by backfill). The lead image is taken from itunes/RSS <image>, then an image <enclosure>, then Media RSS (<media:content> / <media:thumbnail> — how most news feeds, e.g. Fox, ship the article image), then the first inline <img>. At serve time the API rewrites the stored lead-image URL to a signed same-origin /api/img path (see the image proxy under the API surface, and Security › Image proxy) so client-side content/tracker blockers — which match on publisher CDN domains — don't strip it. Body HTML is then de-cluttered: stripCommentsResidue removes aggregator "Comments"/"Read more" residue, and feed.StripPublisherAds removes curated per-publisher in-body sponsored blocks (matched by ad asset/CTA URL since CSS classes are gone after sanitization; a no-op for hosts we haven't vetted). The same cleaning runs in the /extract re-extract path, so existing articles can be re-cleaned on demand.
   • Error → increment error_count, schedule next try with exponential backoff (capped at MaxInterval). This covers parse failures as well as fetch failures: a feed that responds 200 with something gofeed can't parse is just as broken as one that won't connect, and both increment error_count, so both widen the retry interval. (Parse failures used to skip the backoff and keep re-requesting at the floor interval indefinitely.)
4. next_fetch is set by AdaptiveInterval, clamped to [floor, MaxInterval]. The floor is runtime-configurable — EMBER_POLL_MIN_INTERVAL (default 30m) overlaid by the poll_min_interval_seconds app_settings row (admin UI), resolved live per fetch and clamped to the hard bounds store.PollMinInterval{Floor,Ceil} (5m–24h).
5. Newly-inserted articles enqueue on summaryCh (best-effort, drops on full).
6. Filters apply per-subscriber as articles land.

The summary worker is a separate goroutine that drains summaryCh, calls Ollama, writes summary + summary_model + cleaned_html. Failures stamp summary_model = 'skipped' so the article still surfaces in the UI.

summaryCh is bounded, and EnqueueSummary drops on a full queue rather than blocking the poller. A dropped article keeps an empty summary_model, so enqueuePendingSummaries re-queues stragglers every tick — it used to run only at startup, which left a dropped article hidden behind the summary gate until the process restarted.

The summary gate itself is time-boxed rather than absolute: store.summaryGate passes an article that is summarized or was fetched before SummaryGraceBefore (now − summary_grace_seconds, default 120s). All three gate sites — the list/count filter, the cross-feed dedup sibling subquery, and ListFeedsForUser — must use that helper, or a badge disagrees with its column.

Restart safety comes from the same sweep: enqueuePendingSummaries seeds the channel from any article with an empty summary_model, so a restart mid-backlog loses nothing.

Per-feed opt-out (subscriptions.summarize, default 1) is enforced at the queue consumer, in summarizeOne, not at the producers — articles reach summaryCh from poller ingest, the email inbox, resummarize-all, the per-feed re-summarize action, and the every-tick pending sweep, so one check covers them all. A suppressed article is stamped summary_model = 'excluded': the marker is non-empty on purpose, so the row neither re-queues forever (ListUnsummarizedIDs selects on an empty summary_model) nor sits behind the summary gate waiting out a grace window it will never satisfy.

Suppression is unanimous, not per-user: the summary lives on the shared article row, so a feed is skipped only when it has subscribers and none of them want summaries. A feed with no subscribers fails open — handleAddFeed does upsert → subscribe → refresh, and a poller tick can ingest in the window before the first subscription exists. For a subscriber who opted out while others didn't, the summary is blanked per-user at serve time (store.summaryProjection, applied in ListArticles, GetArticleForUser, and Search) rather than removed from storage. Toggling back on runs ResetExcludedByFeed, which clears only the 'excluded' marker (never 'skipped', which belongs to the re-summarize action) and re-enqueues those ids.

Summarizer pipeline

poller.summarize ─> Summarizer.Summarize(title, text)
                       │
                       ├─ Ollama: POST /api/generate
                       │   prompt = labeled "SUMMARY: / POINTS: / CLEANED:"
                       │   options = {temperature, top_p, num_ctx} from app_settings
                       │   timeout = 90s, one retry on transient error
                       │
                       └─> parseResult(s):
                              parseLabeled → parseJSONObject → parseJSONArray
                              → line-based fallback
                              + cleanBullets (strip markers / inline markdown /
                                              prompt-echo / label prefixes)
                              + stripEmphasis (** __ * ` " ')
                              + cleanParagraph

Active model + tunables held in atomic.Value/atomic.Pointer on the Ollama struct so the admin API can swap them without restarting.

Cross-feed dedup

When a wire story appears in multiple feeds a user subscribes to (Reuters and The Verge both run the same headline; a Smashing Magazine post is linked from Hacker News with a different referrer), the article list collapses peers to a single row and offers a click-through to siblings.

Keys. Three columns on articles are populated at ingest and at backfill (internal/feed/canonical.go, internal/feed/fingerprint.go):

• canonical_url — input URL with tracking query params stripped (utm_*, _hs*, mc_*, fbclid, gclid, ref, …), host lowercased, fragment dropped, trailing slash trimmed.
• cluster_id — short hex (8 bytes of SHA-1) over the canonical URL. Used as the equality key; the SHA is non-cryptographic, just a content-addressable hash.
• title_fingerprint — lowercased title with non-alphanumeric runs collapsed to spaces and a ~25-word stopword list dropped. Rejected (empty) below an 8-rune floor so generic titles ("News", "Re:") don't over-collapse.

Predicate. The list query in internal/store/articles.go keeps the lowest-id row in each cluster and filters peers via NOT EXISTS over an OR'd match:

1. Same cluster_id — exact canonical-URL match. Always clusters.
2. Same title_fingerprint within 48h of the candidate's published_at — catches wire stories under different URLs. The window keeps a recurring headline ("Apple Q3 earnings") from collapsing across years.

The suppressor sibling must satisfy the same window, summary gate, muted, and read/star/later predicate as the rows being listed. Otherwise a lower-id copy the user can't see (already read, out of window, unsummarized) would silently suppress its visible copies and collapse a badge to 0 — so a row is only deduped against a peer actually present in the same view.

Only the shared view (explicit one-off share) and board views (explicit curation) skip the predicate. Per-feed and per-category views dedup too: a duplicated story is owned by its lowest-id (first-ingested) feed and counted/shown once. Their sidebar badges go through CountUnreadByFeed / CountUnreadByCategory, which reuse buildArticleFilter, so per-feed badges sum to All Unread and never exceed their column. Rows with both cluster_id and title_fingerprint empty pass unconditionally (no signal in either dimension). Star/later/tag state is per-row and not propagated across siblings (you can save or tag one specific copy). Read state, however, propagates across the cluster: because the suppressor only hides unread copies, marking just the visible winner read would let its hidden sibling resurface as unread on the next poll — a phantom duplicate. So every mark-read path (opening an article, the keyboard read toggle, and bulk "mark all read") routes through MarkReadWithSiblings (the include_siblings flag on POST /api/articles/read), which also marks the cluster's still-unread siblings read. The reverse direction is handled at ingest: a duplicate that arrives after you've read the story — a second feed's copy, or the same feed re-publishing under a fresh guid+content_hash (which UpsertArticle stores as a new row) — would otherwise come in unread with no read winner to suppress it. So UpsertArticle inherits the read state from an already-read cluster sibling at insert time, the ingest-side mirror of MarkReadWithSiblings. Between the two, reading one copy of a syndicated story means you've read the story — and its past and future duplicates stay read.

dup_count + sibling expansion. The list query also returns a dup_count per row: the number of other articles in the user's subscription set that share either match. The SPA renders this as the "Also in N feeds" pill. Clicking the pill calls GET /api/articles/{id}/cluster, which returns the sibling rows (article id, feed id + title, raw URL, per-user read/starred) so the popover can show them.

Backfill. Historical rows inserted before migrations 0013_dedup_canonical.sql / 0014_title_fingerprint.sql start with empty keys. Store.BackfillClustersAsync (kicked off in cmd/ember/main.go) walks them in 500-row batches in a goroutine so it doesn't block startup. The partial indexes (idx_articles_cluster, idx_articles_fp_pub) exclude empty values, so unfilled rows never falsely cluster with each other while the backfill is in flight. Idempotent — after the corpus is full, every restart is a single SELECT that returns zero rows.

SPA

• Svelte 5 with runes ($state, $derived, $effect, $props).
• Vite 5 build, output copied to internal/web/dist, served via embed.FS.
• Typed fetch client in web/src/lib/api.ts (throws ApiError).
• Stores in web/src/lib/stores.ts for user, feeds, categories, boards, articles, themes, branding, new-article counter, etc.
• 15s auto-refresh poll while the tab is visible; SSE not used — REST polling is simpler and fits the cadence. In the Fresh and All Unread views the auto-poll is non-disruptive: newly-arrived articles are held in a backlog (pendingNewIds in stores.ts) and counted once rather than injected into the list under a scrolling reader. They page in on the next full load — loadArticles (which clearNewArticleBacklog resets) — e.g. the refill after "Mark all read". The explicit "Refresh feeds now" control calls pollForNewArticles({ immediate: true }) to bypass the backlog; all other views merge new articles immediately.
• Mark all read in Fresh / All Unread drops the read cards and pages in the next unread batch, with a one-shot grace for the article currently open: that card is kept (greyed via .story.read, still in articles.items so the reader pane keeps it) so you can finish reading; the next "Mark all read" hides it. Tracked by a graceUsedId ledger because opening an article auto-marks it read, so is_read can't distinguish the first click. This top-bar pill marks only the loaded cards. The sidebar's All Unread row also has a hover "Mark all as read" action (markAllUnreadRead in Sidebar.svelte) that instead POSTs { view: "unread" } to mark-all-read, clearing the entire unread set server-side (bounded by UnreadCutoff) regardless of what's paged in. Article-body links are rewritten to target="_blank" rel="noopener noreferrer" after render (lib/links.forceNewTabLinks) so they open in a new tab.
• Optimistic badge updates are window-aware. Marking an article read adjusts the Fresh and All Unread badges locally before the server responds, then a debounced refreshSmartCounts reconciles. Both badges are server-computed over a window, so each optimistic delta must apply that same window — /api/me reports fresh_window_seconds and unread_window_seconds for exactly this. It matters because Starred, Read Later, Shared and board lists are deliberately unwindowed and routinely show older articles: counting one of those toward the All-Unread delta would subtract an article the badge never included. The windows are sent as widths rather than cutoff timestamps, since the server recomputes now - width per request and a timestamp would go stale in a long-lived tab.
• Service worker (web/public/sw.js) caches assets immutably and falls back to cached shell when offline.

Admin endpoints (admin-only)

• GET /api/admin/llm — detected hardware, recommendation, installed models, current model + options.
• POST /api/admin/llm/model / …/pull / …/delete / …/options — switch / pull / delete / tune.
• GET /api/branding (public) / POST /api/admin/branding (admin).
• GET /api/admin/db — size, page count, recent backups + OPML exports, schedules.
• POST /api/admin/db/backup / …/cleanup / …/schedule — manual + scheduled maintenance.
• POST /api/admin/db/opml-export — write an OPML export now (the "Export now" button), honoring the configured export directory + retention.
• DELETE /api/admin/db/backups/{name} / …/exports/{name} — delete one backup or OPML export. {name} is validated to a bare basename with the expected extension, so it can only target a file inside the configured directory (no path traversal).
• POST /api/feeds/resummarize-all — re-process every article after a prompt change.
• GET /api/admin/session / POST /api/admin/session/ttl — server-wide session cookie lifetime.
• GET /api/admin/settings / PATCH /api/admin/settings — SMTP relay config + initial-backlog window + update_check_enabled toggle. Overlays env-derived defaults at runtime; digest sender re-resolves every tick.
• POST /api/admin/settings/email-test — send a one-off diagnostic message through the live SMTP config.

Auth-required (not admin-only):

• POST /api/articles/{id}/extract — on-demand readability re-run for the reader pane's "Re-extract" button. Subject to the same SSRF check as the poller's automatic enrichment.
• GET /api/img?u=…&s=… — same-origin image proxy for article lead images. The API signs the source URL (HMAC keyed off EMBER_SESSION_KEY, domain-separated) when it rewrites image_url in list/detail/search responses, so the endpoint is a capability — it only fetches images Ember itself selected, never arbitrary client input. The outbound fetch runs through urlcheck.Check, accepts only image/*, and is size- (10 MiB) and time- (15 s) bounded. Streams through with a 1-day Cache-Control; no server-side cache.
• GET /api/filters/export / POST /api/filters/import — per-user filter backup. Export downloads the caller's rules as a JSON bundle (ids/timestamps stripped); import validates each rule like a manual create, skips anything invalid or beyond the per-user cap, and is scoped to the calling user (so it's safe to restore on another instance).

E2E

web/e2e/*.spec.ts (Playwright). The binary in EMBER_TEST_MODE=1 seeds a deterministic admin + 1 feed + 12 articles with known content, so specs can assert on titles/excerpts directly. Axe-core runs against the major screens for WCAG 2.1 AA.