Serve the status page from Redis, and keep the alarm honest (#149)

/api/crawlstats answered in 118 seconds. It fans out eleven reads and returns
when the slowest does; timed against production:

    categoryStats            30,005ms  (timed out)
    jobBacklogs              11,255ms
    failingFeeds(20)          5,172ms
    crawlStats                4,975ms
    the other seven          under 600ms each

No rewrite fixes `categoryStats`. It is a group-by over 476,715 rows, and on
the same connection a bare `count(*)` of that table is 6.9s while `select
category, count(*) … group by category` does not finish inside the client's 30s
deadline. Dropping its conditional aggregates -- the fix that worked for
`crawlStats` in PR #96 -- changes nothing, because the cost is visiting every
row for a column no index covers.

The per-process cache that was already here could not save it either, for a
reason worth naming: `categoryStats` does not run slowly, it *fails*, and a
cache that only stores successes stores nothing. Every request paid the full
timeout, for ever. Redis plus serve-stale-on-failure inverts that -- one
success, any time, serves every later reader -- and it survives the deploys that
emptied the old cache. It also adds no writes to Turso, whose write path is the
binding constraint on everything else here.

The part that needed care is that a status page must never report a stalled
crawler as healthy. The rule that keeps it honest is to cache facts and derive
anything measured against now: `idleMinutes` is `now - lastSuccessAt` computed
inside the query, so caching the object freezes it, and a dead crawler would go
on reporting the same cheerful number. `liveStats` caches the timestamp and
redoes the subtraction, so the number climbs while the crawler is down.
`queueHistory` does the same with its hour labels, caching the sparse rows and
filling the window on the way out.

A cache that can hang is not a cache, so the lookups are bounded too, and every
failure path -- no REDIS_URL, a refused connection, a socket that accepts
commands and never answers -- falls through to the read it replaced.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: ralyodio

apps/web/src/app/api/crawlstats/route.js

@@ -1,7 +1,14 @@
-import { q, discovery, alerts } from '@rssamplifier/db';
+import { q, discovery } from '@rssamplifier/db';
 
 import { db } from '../../../lib/db.js';
-import { categoryStats, indexingHistory, jobBacklogs } from '../../../lib/crawlstats.js';
+import {
+  categoryStats,
+  indexingHistory,
+  jobBacklogs,
+  liveStats,
+  failingFeeds,
+  alertingAccounts,
+} from '../../../lib/crawlstats.js';
 import { toLine } from '../../../lib/crawlLog.js';
 import { jobRows } from '../../../lib/jobs.js';
 
@@ -15,10 +22,15 @@ export const dynamic = 'force-dynamic';
  * backlog that should drain within a tick or two, `stale` is active feeds the
  * crawler has not successfully read in a day.
  *
- * Deliberately uncached — a status endpoint that answers from a cache reports
- * that everything was fine a minute ago, which is the one thing it must not do.
- * The two additions that are cached, briefly, are the ones nothing would alert
- * on: the hourly history and the category breakdown. See lib/crawlstats.js.
+ * Every read here is cached, and the distinction that keeps that honest is
+ * between a fact and a derivation. A count is a fact and may be ten seconds
+ * old; `idleMinutes` is `now - lastSuccessAt`, so caching it would freeze the
+ * one number a monitor alerts on. `liveStats` caches the timestamp and redoes
+ * the subtraction — see lib/crawlstats.js.
+ *
+ * Before this, the endpoint answered in 118 seconds: `categoryStats` no longer
+ * completes inside the client's 30s deadline, and since a cache that only
+ * stores successes never stored it, every request paid the full timeout.
  */
 export async function GET() {
   const client = db();
@@ -36,8 +48,8 @@ export async function GET() {
     alertAccounts,
     operationalErrors,
   ] = await Promise.all([
-    q.crawlStats(client),
-    q.failingFeeds(client, 20),
+    liveStats(),
+    failingFeeds(),
     q.recentlyCrawled(client, 20),
     discovery.countQueuedCandidates(client),
     discovery.countQueuedKeywords(client),
@@ -57,7 +69,7 @@ export async function GET() {
     q.logActivity(client, 1),
     // See the page: this only tells a sender with nobody to serve from one that
     // has stopped, which the log alone cannot say.
-    alerts.alertingAccountCount(client),
+    alertingAccounts(),
     q.crawlOperationalErrors(client, { limit: 20, hours: 24 }),
   ]);
 

apps/web/src/app/crawlstats/page.jsx

@@ -1,11 +1,14 @@
-import { q, discovery, alerts } from '@rssamplifier/db';
+import { q, discovery } from '@rssamplifier/db';
 
 import { db } from '../../lib/db.js';
 import {
   categoryStats,
   indexingHistory,
   jobBacklogs,
   queueHistory,
+  liveStats,
+  failingFeeds,
+  alertingAccounts,
   GROWTH_DAYS,
 } from '../../lib/crawlstats.js';
 import { describe, toLine } from '../../lib/crawlLog.js';
@@ -62,8 +65,8 @@ export default async function CrawlStatsPage() {
     operationalErrors,
     queues,
   ] = await Promise.all([
-    q.crawlStats(client),
-    q.failingFeeds(client, 50),
+    liveStats(),
+    failingFeeds(50),
     q.recentlyCrawled(client, 15),
     discovery.countQueuedCandidates(client),
     discovery.countQueuedKeywords(client),
@@ -90,7 +93,7 @@ export default async function CrawlStatsPage() {
     // Only to tell a sender with nothing to do from one that has stopped: the
     // alert pass writes no log line at all when nobody is subscribed, and a
     // silent job is otherwise indistinguishable from a dead one.
-    alerts.alertingAccountCount(client),
+    alertingAccounts(),
     // Kept separately from the rolling live-log window. At crawler throughput,
     // 400 successful feed lines can evict an operational failure in minutes.
     q.crawlOperationalErrors(client, { limit: 20, hours: 24 }),