Believe a 429 once for the whole host, instead of once per feed

Reported as the crawler being slow again: 5,000 feeds/hour down to 2,000. It was
neither the database nor a knob, and the diagnostic order in the notes settled
that in three minutes — an empty write transaction measured 0.39s (the old
saturation was 29-300s), and the poller's own `started` event showed all four
tuning variables already correct.

The live log said it instead. 130 of 136 feed events in five minutes were
`www.reddit.com`, and 106 of those were http-429.

**A bulk import put 50,099 reddit feeds in the directory** — 30,000 on 08-28 and
19,999 on 08-27, two submissions, which is exactly when throughput halved.
49,174 of them were due at once: 41% of the entire due queue on a single
hostname. Over one hour, reddit took 1,074 crawl attempts and was refused 1,015
times, while the whole rest of the directory got 218. The crawler had not slowed
down. It was spending 84% of itself being told to go away, and had read 650 of
those 50,099 feeds successfully in 36 hours of trying.

Two things combine to make one host able to do that, and neither is wrong on its
own. A host's feeds are crawled strictly in series — that is the politeness
guarantee. And `spreadHosts` deliberately lifts its per-host cap rather than
under-fill a batch, because the alternative throttles a legitimate bulk import
to eight feeds a tick. So when the due set genuinely is one host, one worker
gets most of a 900-feed batch on that host and walks it into the same rate limit
nine hundred times.

**The fix is to stop asking.** A 429 is a fact about the host, not about the feed
that happened to be at the front of its queue, so the worker now believes it
once and leaves. The rest of that host's queue is put back rather than walked.

Three things that decide whether this is right:

  * **The held-back feeds are not blamed.** `markThrottled` already exists so a
    rate limit is not recorded as feed health — ten consecutive failures marks a
    feed dead, so counting 429s against feeds would retire a whole platform for
    our own crawl rate. `markHostThrottled` extends that to feeds we did not
    even send a request for, where there is even less to judge.

  * **They come back spread across an hour, not at one instant.** Handing a
    thousand held-back feeds a single timestamp just moves the pile-up one
    interval into the future, where it walks into the same wall together.

  * **It is bucketed by minute, not one statement per feed.** This database gives
    the cluster one writer, and a batch of nine hundred statements holds it long
    enough to push the jobs behind it past their deadline — the failure
    `TURSO_QUEUE_GROUP_STATEMENTS` was lowered to 10 to avoid. Bucketing makes
    the statement count a property of the window (~60) rather than of the queue.

A 404 still walks the queue to the end: that is evidence about one feed and says
nothing about the host.

`crawlDue` gained a `crawl` option so a whole batch can be driven without the
network — the throttle path only exists *across* a host's queue, so it was not
reachable from `crawlFeed`'s tests at all. The tick summary gains `throttled`,
which is what distinguishes a slow tick that was one publisher rate-limiting us
from the crawler struggling; `ms` alone cannot tell those apart.

Also done, in production rather than in code: the 50,099 reddit feeds were
rescheduled across fourteen days, taking that host from 49,174 feeds due at once
to ~145 an hour. Nothing was deleted and the update only ever pushed schedules
out. Whether a directory that describes itself as the small web rather than the
platforms should carry 50k subreddits at all is a separate question, and not one
to answer by deleting someone's upload.

Suite green: 1,206 tests, 0 failures across 11 packages.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014gV7AyoWsiihaakh7RYZbP

Author: ralyodio

apps/poller/src/index.js

@@ -289,7 +289,7 @@ async function tick() {
     // stdout: they are the content of a log somebody is watching, and twenty-five
     // a minute forever is not what Railway's viewer is for. The tick summary
     // below is the durable record of the same work.
-    const { crawled, failed, items, unchanged, hosts } = await crawlDue(
+    const { crawled, failed, items, unchanged, throttled, hosts } = await crawlDue(
       db,
       batchSize,
       concurrency,
@@ -309,6 +309,12 @@ async function tick() {
         // entirely up to that server, so this cannot be predicted from here and
         // has to be watched.
         unchanged,
+        // Hosts that answered 429 and had the rest of their queue put back
+        // rather than walked into the same wall. This is the number that says a
+        // slow tick was one publisher rate-limiting us and not the crawler
+        // struggling — the two are indistinguishable from `ms` alone, and
+        // telling them apart is what this whole change is for.
+        throttled,
         // How many distinct hosts the batch touched. The number that says
         // whether the worker pool had anything to do: `crawled` and `ms`
         // together look identical for a batch spread over 80 hosts and one

packages/db/src/queries.js

@@ -3022,6 +3022,70 @@ export async function markThrottled(db, id, minutes) {
   });
 }
 
+/**
+ * Come back later for every feed still queued on a host that just throttled us.
+ *
+ * `markThrottled` answers the one feed that was refused. This answers the rest
+ * of that publisher's queue, and it exists because the one-feed version is the
+ * wrong shape of reply to a 429: a throttle is a fact about the *host*, not
+ * about the feed that happened to be at the front of it. Measured on
+ * 2026-08-29, a bulk import put 50,099 feeds on `www.reddit.com` — 41% of the
+ * whole due queue — and because a host's feeds are crawled strictly in series,
+ * one worker walked that queue into the same 429 over and over: 84% of every
+ * crawl attempt in an hour went to reddit and 1,015 of 1,074 were refused,
+ * while the rest of the directory got 218 crawls. Stopping at the first refusal
+ * turns that whole wasted walk into a single request.
+ *
+ * **The spread is the point, not a detail.** Rescheduling a thousand held-back
+ * feeds to one instant just moves the pile-up, and they would come due together
+ * and walk into the same wall in one tick's time. Each feed is given its own
+ * offset across `spreadMinutes`, so the host's queue returns as a trickle that
+ * its rate limit can actually absorb.
+ *
+ * Like `markThrottled`, only the schedule moves: `status`, `error_count`,
+ * `last_error`, `last_success_at` and `last_fetched_at` are all left alone.
+ * These feeds were never even asked, so there is nothing to judge them on —
+ * recording a failure here is how a rate limit would retire a whole platform.
+ *
+ * @param {Client} db
+ * @param {string[]} ids feeds left unread on the throttled host
+ * @param {number} minutes how long before the first of them is tried again
+ * @param {number} [spreadMinutes] window to scatter them across
+ * @returns {Promise<number>} how many were rescheduled
+ */
+export async function markHostThrottled(db, ids, minutes, spreadMinutes = 60) {
+  const queued = (Array.isArray(ids) ? ids : []).filter((id) => typeof id === 'string' && id);
+  if (queued.length === 0) return 0;
+
+  const wait = Math.max(1, Math.round(Number(minutes) || 30));
+  const spread = Math.max(0, Math.round(Number(spreadMinutes) || 0));
+  const now = nowIso();
+
+  // Bucketed by minute rather than one statement per feed, and that is a
+  // constraint rather than a tidiness preference. A throttled host's queue here
+  // runs to hundreds of feeds, and this database gives the whole cluster one
+  // writer: a batch of nine hundred statements holds it long enough to push the
+  // jobs behind it past their deadline, which is the failure
+  // `TURSO_QUEUE_GROUP_STATEMENTS` was lowered to 10 to avoid. Bucketing makes
+  // the statement count a property of the *window* -- at most `spread` + 1,
+  // sixty-odd -- no matter how many feeds are held back.
+  const buckets = new Map();
+  for (let index = 0; index < queued.length; index += 1) {
+    const offset = spread === 0 ? 0 : Math.round((index / queued.length) * spread);
+    const bucket = buckets.get(offset);
+    if (bucket) bucket.push(queued[index]);
+    else buckets.set(offset, [queued[index]]);
+  }
+
+  const statements = [...buckets.entries()].map(([offset, members]) => ({
+    sql: `update feeds set next_fetch_at = ?, updated_at = ? where id in (${members.map(() => '?').join(',')})`,
+    args: [nowIso((wait + offset) * 60_000), now, ...members],
+  }));
+
+  await db.batch(statements, 'write');
+  return queued.length;
+}
+
 /* ------------------------------------------------------------- feed cards */
 
 /**

packages/ingest/src/crawl.js

@@ -228,7 +228,15 @@ export async function crawlFeed(db, feed, opts = {}) {
   // So: come back when asked, leave every health column exactly as it was.
   if (resolved.throttled) {
     await q.markThrottled(db, id, throttleMinutes(resolved.retryAfter));
-    return { ok: false, newItems: 0, throttled: true, error: resolved.error };
+    // `retryAfter` travels with the result so the caller can hold the rest of
+    // this host's queue back by the same interval the server itself named.
+    return {
+      ok: false,
+      newItems: 0,
+      throttled: true,
+      retryAfter: resolved.retryAfter ?? null,
+      error: resolved.error,
+    };
   }
 
   if (!resolved.ok) {
@@ -575,7 +583,9 @@ function hostOf(feed) {
  * @param {number} [batchSize]
  * @param {number} [concurrency] hosts crawled at once
  * @param {((event: { at: string, event: 'feed', status: 'ok'|'error', subject: string, slug: string|null, amount: number|null, detail: string|null, ms: number }) => void)|null} [onEvent]
- * @param {{ perHost?: number }} [opts]
+ * @param {{ perHost?: number, crawl?: { resolve?: Function, scrape?: Function } }} [opts] `crawl` is
+ *   handed to each `crawlFeed`, which is what makes a whole batch testable without
+ *   the network — the throttle path in particular only exists across a host's queue.
  * @returns {Promise<{ crawled: number, failed: number, items: number, hosts: number }>} items being posts stored, not posts seen
  */
 export async function crawlDue(db, batchSize = 25, concurrency = 8, onEvent = null, opts = {}) {
@@ -604,6 +614,9 @@ export async function crawlDue(db, batchSize = 25, concurrency = 8, onEvent = nu
   // out there -- it is entirely up to other people's servers, so it cannot be
   // predicted and has to be measured.
   let unchanged = 0;
+  // Hosts abandoned mid-queue because they answered 429. Distinct from `failed`:
+  // nothing is wrong with these feeds and they were mostly never even asked.
+  let throttled = 0;
   let next = 0;
 
   const worker = async () => {
@@ -612,14 +625,17 @@ export async function crawlDue(db, batchSize = 25, concurrency = 8, onEvent = nu
       next += 1;
       if (index >= queues.length) return;
 
-      for (const feed of queues[index]) {
+      const queue = queues[index];
+
+      for (let position = 0; position < queue.length; position += 1) {
+        const feed = queue[position];
         const started = Date.now();
 
         // One feed that throws — a write that times out, a URL that breaks the
         // parser — must not reject the whole batch and take the other workers'
         // completed crawls down with it.
         try {
-          const res = await crawlFeed(db, feed);
+          const res = await crawlFeed(db, feed, opts.crawl);
           if (res.ok) {
             crawled += 1;
             items += Number(res.newItems ?? 0);
@@ -631,6 +647,20 @@ export async function crawlDue(db, batchSize = 25, concurrency = 8, onEvent = nu
             amount: res.ok ? Number(res.newItems ?? 0) : null,
             detail: res.ok ? null : (res.error ?? 'unknown'),
           });
+
+          // A 429 is the host speaking for all of its feeds, so believe it once
+          // and leave. The rest of this queue is the same server, and asking it
+          // the same question another eight hundred times is precisely what it
+          // just told us to stop doing — see `markHostThrottled` for the
+          // measurement that put this here.
+          if (res.throttled) {
+            throttled += 1;
+            const rest = queue.slice(position + 1);
+            if (rest.length > 0) {
+              await holdBackHost(db, rest, throttleMinutes(res.retryAfter), onEvent, feed);
+            }
+            break;
+          }
         } catch (err) {
           failed += 1;
           report(onEvent, feed, started, {
@@ -649,7 +679,67 @@ export async function crawlDue(db, batchSize = 25, concurrency = 8, onEvent = nu
   // `hosts` is what says whether the spread is working: a batch of 300 feeds
   // across 4 hosts cannot go faster than its biggest queue however many workers
   // are pointed at it, and the number is otherwise invisible from outside.
-  return { crawled, failed, items, unchanged, hosts: queues.length };
+  return { crawled, failed, items, unchanged, throttled, hosts: queues.length };
+}
+
+/**
+ * Put back the feeds a throttled host never got asked about.
+ *
+ * Separated from the worker so the worker reads as the policy — believe the
+ * 429, leave — rather than as the bookkeeping. Two things it must not do, and
+ * both are why it exists as its own function:
+ *
+ *   * it must not fail the batch. These feeds are fine; the reschedule is an
+ *     optimisation and the crawl has already done its useful work. If the write
+ *     times out they keep their existing `next_fetch_at` and are simply offered
+ *     again next tick, which is the behaviour this replaces.
+ *   * it must not report the held-back feeds as errors. Nothing was wrong with
+ *     them and nothing was even sent, so a per-feed line would put hundreds of
+ *     healthy feeds on the failure panel. One line for the host says it.
+ *
+ * @param {import('@libsql/client').Client} db
+ * @param {Array<{ id: string }>} rest feeds left unread on this host
+ * @param {number} minutes how long before the first is tried again
+ * @param {((event: object) => void)|null} onEvent
+ * @param {{ feed_url: string, title?: unknown }} feed the one that was refused
+ * @returns {Promise<void>}
+ */
+async function holdBackHost(db, rest, minutes, onEvent, feed) {
+  const started = Date.now();
+
+  try {
+    // Spread across an hour rather than returned all at once: a thousand feeds
+    // handed back to the same instant is the same pile-up one tick later.
+    await q.markHostThrottled(
+      db,
+      rest.map((f) => f.id),
+      minutes,
+      60,
+    );
+  } catch {
+    // The schedule is unchanged, so they come back next tick exactly as they
+    // would have without this. Losing the batch over it would be worse.
+    return;
+  }
+
+  if (typeof onEvent !== 'function') return;
+
+  try {
+    onEvent({
+      at: new Date().toISOString(),
+      event: 'host-throttled',
+      status: 'info',
+      subject: hostOf(feed),
+      slug: null,
+      amount: rest.length,
+      // Not `message`: `toEntry` reads any row carrying one as an error, and
+      // this is the crawler behaving correctly. See the daemon error panel note.
+      detail: `held back ${rest.length} feeds for ${minutes}m`,
+      ms: Date.now() - started,
+    });
+  } catch {
+    // A broken listener loses its line and nothing else.
+  }
 }
 
 /**