A homepage must not 500 because a count is slow

The site was answering `/` with a 500 every few minutes. Measured 2026-08-23:
30.1s then a 500, thirteen seconds into a `stats-warm` window; 5.5s on the
retry; 0.24s once the window passed. Three things were true at once and this
changes all three.

**The warmer was running at a 42% duty cycle.** `statsTick` runs `categoryStats`
-- nine conditional aggregates over 476,726 feeds, which no index covers -- on
a connection given a deliberate 150s deadline, every `STATS_WARM_SECONDS` and
that defaults to 300. Over 274 minutes: 56 runs, 86-150s each, 42.4% of the
wall clock, and 20 of the 56 abandoned at the ceiling having paid the whole
scan for nothing. `STATS_WARM_SECONDS=3600` is now set on the poller service.
The app already serves this blob up to a day stale, so hourly is honest.

**A read is not free, whatever the comment says.** Both `statsTick` and
`statsWarmer.js` justify the job with "it is a read, and reads do not queue
behind the single writer everything else contends for". That is true of
`serializeWrites` and the Redis queue, which are in-process, and false at the
database: one Turso instance, one SQLite file. Probed from outside across a
confirmed 02:25:41-02:26:39 window, reads held at 79ms while an *empty* write
transaction took 55,699ms -- one call spanning the entire warm and returning as
it ended. That is where the crawler's `write-retried` -> `write-retried` ->
`write-failed` chain comes from, and it is ~90s wide because the client
deadline is 30s and it tries three times.

**The web service was starting the same scan the poller had just given up on.**
`CATEGORY_TTL_MS` was five minutes, so past that `remember` fires a background
refresh, and on this key the refresh is the 86-150s scan against a 20s timeout
it has never once met. Every time the warmer was abandoned the entry went stale
and a reader picked the scan back up. Now an hour, matched to the warmer.

**And the homepage had no guard at all.** It called `listFeeds`, `countFeeds`
and `countFeedsByKind` directly, uncached, with no deadline and no fallback.
`crawlstats.js` already documents what two of those cost at this size: a bare
`count(*)` of `feeds` is 6.9s, and `select category, count(*) ... group by
category` -- which is `countFeedsByKind` verbatim -- exceeds the 30s client
deadline. When it does, nothing catches it and Next answers the site's most
visited URL with a 500. They now go through `remember` in one key, so a stalled
database costs a slightly old count instead of an error page, and a burst of
readers costs one scan rather than one each.

The interval change removes most of the occasions. The homepage change is the
half that matters when there is a next one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: ralyodio

apps/web/src/app/page.jsx

@@ -1,6 +1,5 @@
-import { q } from '@rssamplifier/db';
-
-import { db, siteUrl } from '../lib/db.js';
+import { siteUrl } from '../lib/db.js';
+import { directoryIndex } from '../lib/directory.js';
 import { feedAlternates } from '../lib/subscribe.js';
 import { AD_TEXT, adPlan } from '../lib/ads.js';
 import Ad from './Ad.jsx';
@@ -46,12 +45,10 @@ export const metadata = {
  * Directory index: newest blogs first, with the submit box up top.
  */
 export default async function Home() {
-  const client = db();
-  const [rows, total, byKind] = await Promise.all([
-    q.listFeeds(client, { limit: 60 }),
-    q.countFeeds(client),
-    q.countFeedsByKind(client),
-  ]);
+  // Cached in Redis and served stale on failure. Two of these three reads are
+  // whole-table work at half a million feeds, and uncached they answered this
+  // URL with a 500 whenever the database was busy — see ../lib/directory.js.
+  const { rows, total, byKind } = await directoryIndex();
 
   // The index is a long scan, so ads go *between* rows rather than around the
   // list. First one is deep enough that the fold is all directory, and there

apps/web/src/lib/crawlstats.js

@@ -42,8 +42,29 @@ import { db } from './db.js';
 /** How long an hourly rollup read is trusted. */
 const HISTORY_TTL_MS = 60 * 1000;
 
-/** How long a category breakdown is trusted. */
-const CATEGORY_TTL_MS = 5 * 60 * 1000;
+/**
+ * How long a category breakdown is trusted.
+ *
+ * Matched to the poller's `STATS_WARM_SECONDS`, and that pairing is the whole
+ * point rather than a coincidence. Past this age `remember` fires a background
+ * `refresh()`, and on this key the refresh is a `categoryStats` full scan that
+ * takes 86–150 seconds against half a million feeds — it cannot finish inside
+ * `CHART_TIMEOUT_MS` and never once has. So a reader arriving after the TTL
+ * lapsed was starting a doomed 20-second scan on the one instance the warmer
+ * was already scanning, and throwing the result away. (`remember` dedupes to
+ * one in-flight refresh per key, so this was one wasted scan at a time, not
+ * one per reader — still one too many.)
+ *
+ * Five minutes made that certain: the warmer is abandoned at its own 150s
+ * ceiling roughly a third of the time, and each time it was, the entry went
+ * stale and the web service picked up the same scan the poller had just given
+ * up on. An hour means the warmer refreshes it before a reader ever asks.
+ *
+ * Safe because the value is served stale for a day anyway
+ * (`CHART_MAX_STALE_MS`): the honest ceiling on how old this may get was never
+ * the TTL.
+ */
+const CATEGORY_TTL_MS = 60 * 60 * 1000;
 
 /**
  * How stale a breakdown may get before a reader waits for a fresh one.

apps/web/src/lib/directory.js

@@ -0,0 +1,113 @@
+import { q, remember } from '@rssamplifier/db';
+
+import { db } from './db.js';
+
+/**
+ * The three reads behind the directory index, cached in Redis.
+ *
+ * ## Why this exists
+ *
+ * The homepage used to call `listFeeds`, `countFeeds` and `countFeedsByKind`
+ * directly in a `Promise.all`, uncached, with no deadline and no fallback. Two
+ * of the three are whole-table work at 476,000 feeds, and this module's sibling
+ * `crawlstats.js` already documents what they cost: a bare `count(*)` of
+ * `feeds` is 6.9 s, and `select category, count(*) … group by category` — which
+ * is `countFeedsByKind`, verbatim — exceeds the client's 30 s deadline. When it
+ * does, the read throws, nothing catches it, and Next answers the site's most
+ * visited URL with a 500.
+ *
+ * That is not hypothetical and it is not only a bad-day failure. Measured
+ * 2026-08-23: `GET /` returned 500 after 30.1 s, thirteen seconds into a
+ * `stats-warm` window, then 5.5 s, then 0.24 s once the window passed. The
+ * warmer's own scan is what pushes these over the line, so the interval change
+ * that goes with this commit removes most of the occasions — but a page whose
+ * only behaviour when the database is slow is to 500 will find the next one.
+ *
+ * ## Why a cache and not a faster query
+ *
+ * Same trade `crawlstats.js` settled: the columns are `category` and `status`,
+ * `status` is rewritten on every crawl, and an index covering it would be paid
+ * for on the write path, which is the binding constraint on this database.
+ * Buying a fast homepage with a slower crawler is the wrong way round.
+ *
+ * ## Why one key and not three
+ *
+ * They are rendered together and they are read together, so one Redis round
+ * trip beats three, and the three numbers can never disagree about which
+ * moment they describe — a total that does not match the sum of the per-kind
+ * counts is the kind of thing a reader notices and nobody can reproduce.
+ *
+ * ## What the reader sees when it fails
+ *
+ * `remember` serves the last good answer for up to a day rather than throwing,
+ * so a stalled database costs a slightly old count instead of an error page.
+ * With nothing cached at all it returns the zero shape below, and the homepage
+ * already renders that: it has an explicit empty-directory branch, and the
+ * shell around it — search, the submit box, the category index — is worth
+ * serving on its own. An empty index for one request beats a 500 for a crawler
+ * that will remember it.
+ */
+
+/**
+ * How long the index is trusted before a refresh is started behind the reader.
+ *
+ * A minute, matching `HISTORY_TTL_MS` next door. "Recently added" is a list
+ * that changes when the crawler admits a feed, not something a visitor is
+ * watching tick over, and a burst of readers should cost one scan rather than
+ * one each.
+ */
+const INDEX_TTL_MS = 60 * 1000;
+
+/**
+ * How stale it may get before a reader waits for a fresh one.
+ *
+ * A day, for the reason `CHART_MAX_STALE_MS` is a day: when the underlying read
+ * is failing the alternative is not a fresher page, it is no page.
+ */
+const INDEX_MAX_STALE_MS = 24 * 60 * 60 * 1000;
+
+/**
+ * Shorter than the client's own 30 s deadline, so a read that is going to hang
+ * gives the page back before the browser gives up on it.
+ */
+const INDEX_TIMEOUT_MS = 20 * 1000;
+
+/** How many blogs the index lists. */
+export const INDEX_LIMIT = 60;
+
+/** What a reader gets when nothing has ever been cached and the read fails. */
+const EMPTY = { rows: [], total: 0, byKind: {} };
+
+/**
+ * Newest blogs, the directory total, and the per-category counts.
+ *
+ * @returns {Promise<{ rows: object[], total: number, byKind: Record<string, number> }>}
+ */
+export async function directoryIndex() {
+  const value = await remember(
+    'directoryIndex',
+    {
+      ttlMs: INDEX_TTL_MS,
+      maxStaleMs: INDEX_MAX_STALE_MS,
+      timeoutMs: INDEX_TIMEOUT_MS,
+      fallback: null,
+    },
+    async () => {
+      const client = db();
+      const [rows, total, byKind] = await Promise.all([
+        q.listFeeds(client, { limit: INDEX_LIMIT }),
+        q.countFeeds(client),
+        q.countFeedsByKind(client),
+      ]);
+      return { rows, total, byKind };
+    },
+  );
+
+  // A cached entry predating a shape change, or the fallback, must not be able
+  // to take the page down a second way — the caller destructures all three.
+  return {
+    rows: Array.isArray(value?.rows) ? value.rows : EMPTY.rows,
+    total: Number(value?.total ?? 0),
+    byKind: value?.byKind && typeof value.byKind === 'object' ? value.byKind : EMPTY.byKind,
+  };
+}