Ask the callers with no name to do some arithmetic

[ralyodio]

The next rung after #186. That one made the site survive the fleet by using the cores it was already paying for; this one makes the fleet pay something.

Off by default. CHALLENGE_ENABLED must be 1 and CHALLENGE_SECRET must be set. Merging changes nothing until both are.

Why the rungs below this one do not reach

Every limit the site had is keyed on who is asking: crawlThrottle meters a caller by address and UA token, tiers.js sorts them onto rungs, the gateway's chargeSpoofedBrowsers asks whether a thing claiming Chrome sends the headers Chrome cannot suppress. All three lose to the same move, and it is the move the traffic makes — ask from somewhere else, every time.

From the 2026-09-07 outage sample: 500 requests in 3.5 s, 500 distinct addresses, no address twice, two Chrome strings between them, and all 500 paths different across 329 topic slugs.

• A per-caller limit sees 500 callers with one request each and refuses nobody.
• A cache is never asked the same question twice and hits nothing.
• The fleet already sends Sec-Fetch-Mode, so the spoof check waves it through.

So this rung is keyed on what the caller can do. Before an expensive page renders, an anonymous caller must find a number whose SHA-256 opens with 18 zero bits. Verifying is one hash. The answer is bound to the address that solved it, which is what the rotation cannot route around: five hundred addresses now means five hundred solves.

What it is honestly worth

Not a wall. A real JavaScript engine can pay this, so against headless browsers it is a tax rather than a refusal — one solve per address per hour instead of per request. What it stops outright is the much larger population replaying headers with no engine behind them. workers.js makes the same point about capacity. The two together are a slope, not a gate, and the module says so where someone will read it.

Who is never asked

crawl-gateway.js's exempt() is reused, not copied, so the two can never drift about who is welcome: a reader with a session, a program with an API key, a named retrieval or search crawler. Training crawlers never reach this code — the gate answers them 402 first. Three more exclusions, each of which would be a bug rather than a policy:

• The API is never challenged. apiguard.js already argues that case and is right. A scraper pushed onto /api/topics/* reads 5 KB of JSON instead of rendering 47 KB of HTML — the trade we want.
• Feed exports are never challenged (.rss, .atom, .json, .opml, .m3u, .pls). A challenge there breaks every subscriber silently and permanently.
• Only /topics/*, /authors/* and */read — 400 of the 500 sampled requests. Everything else is too cheap to interrupt anyone for.

Two things the measurements changed

The hash is written out rather than called through crypto.subtle. A promise per hash dominates: Web Crypto managed ~67k hashes/s end to end, making 18 bits an eight-second wait. The first hand-written version was worse at 17k/s, because applying SHA-256's padding through a function call per byte costs ~1,000 calls per block. The solver now owns the padding and rebuilds it on the ten occasions the nonce gains a digit → 610k hashes/s, so 18 bits is ~0.43 s.

The benchmark had to leave the vm sandbox. Typed arrays across a vm boundary defeat the JIT — the same text does 610k hashes/s under new Function and 10k inside vm.runInNewContext. A sandboxed benchmark would have condemned a solver that is fast in the browser it is written for. The test now compiles it the way a browser does, and says why.

Verification

• node --test test/*.test.js — 425 pass, 0 fail (22 new).
• pnpm build — green.
• The shipped hash text is run in the test and checked against Web Crypto at every padding edge (0, 55, 56, 63, 64, 65, 200 bytes). A hash wrong in one bit would produce a challenge nobody can ever solve, and the only symptom would be readers stuck forever while the logs said nothing.
• End to end against a real server, with the page's own script lifted out of the response verbatim:

1. GET /topics/bing-ai            -> 503 (5,708 bytes, vs a 35,714-byte render)
2. solved 18 bits in 106ms
3. GET /topics/bing-ai + token    -> 200 (35,714 bytes)
4. same token, another address    -> 503   (not transferable)
5. /authors/... with same token   -> 200   (solve once, then browse)

• And the exclusions, live: /api/topics/nfl-season, /topics/bing-ai.rss, / and /login all answered 200 with the feature on.

Turning it on

CHALLENGE_SECRET=<random>     # required; no default, because a shared constant is forgeable
CHALLENGE_ENABLED=1
CHALLENGE_BITS=18             # optional, 1-32
CHALLENGE_TTL_MINUTES=60      # optional, 1-1440

Junk in any of them falls back to the default rather than to zero — a typo must not quietly remove the difficulty while leaving the interruption in place. To turn it off, unset CHALLENGE_ENABLED.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KPvk8mVEpxWwTRFby9m8VT