fix(#751,#752): retention counted deploys, and 30 of them was 3.5 days

.github/workflows/deploy.yml

@@ -201,26 +201,32 @@ jobs:
             "${SITE_URL:-https://scripthammer.com}"
         env:
           SITE_URL: ${{ vars.NEXT_PUBLIC_SITE_URL }}
-          # How many generations back an asset is carried. Bounds _next/static so
-          # chaining cannot grow forever.
+          # HOW LONG an asset is carried. Stated in days because the exposure is a
+          # duration: how long a visitor may still be holding a document.
           #
-          # WAS 5, AND THE REASONING WAS WRONG. It was sized against "deploys per 10
-          # minutes" — the HTML cache-control window. But the thing being protected
-          # is not a 10-minute window, it is HOW LONG SOMEONE'S TAB HAS BEEN OPEN,
-          # and those are unrelated quantities. On 2026-08-09 production rendered
-          # unstyled for the SIXTH time (#438, #467, #476, #548): five deploys landed
-          # across 22 hours — well-paced by the 10-minute rule — and a visitor
-          # returning the next day asked for CSS that had just aged out. Retention
-          # worked exactly as designed; the design measured the wrong thing.
+          # IT COUNTED DEPLOYS TWICE, AND WAS WRONG BOTH TIMES (#751). 5 was sized
+          # against "deploys per 10 minutes" — the HTML cache-control window — and
+          # #650 correctly identified that as the wrong quantity when production went
+          # unstyled a SIXTH time. But its fix kept counting deploys: 30, justified as
+          # "a normal working week even at an unusually high merge rate". Then 40
+          # deploys landed in the following 6 days (19 in one day), making 30
+          # generations ~3.5 days, and production went unstyled an EIGHTH time on
+          # 2026-08-15. Retention worked exactly as designed, twice; the design
+          # measured a quantity nobody had measured, twice.
           #
-          # 30 covers a normal working week even at an unusually high merge rate.
-          # Measured cost: 3-5 hashed files per generation (17 files bought the old
-          # 5), so ~100 small files against a 131-page export.
+          # A duration cannot be invalidated by the merge rate. Two weeks covers a
+          # holiday-length absence. Expected cost at the measured ~7 deploys/day:
+          # ~300-500 small files against a 131-page export (17 files bought the old
+          # 5 generations, ~100 bought 30). RETAIN_MAX_FILES backstops a runaway.
+          #
+          # `check-retained-assets.mjs` in smoke.yml FAILS if the live window is
+          # narrower than this. That assertion is the part that was missing — nothing
+          # ever compared the promise against its intended width.
           #
           # This is a MITIGATION. The recovery that does not depend on any number is
           # the self-heal in src/app/layout.tsx; the actual fix is #635 — a CDN
           # serving HTML `no-cache`, which GitHub Pages cannot be configured to do.
-          RETAIN_GENERATIONS: '30'
+          RETAIN_DAYS: '14'
 
       - name: Upload artifact
         uses: actions/upload-pages-artifact@v3

CLAUDE.md

@@ -423,33 +423,47 @@ The rules below still describe `e2e.yml` and remain correct for it.
 
 ### Merging several PRs in quick succession can serve production with no CSS
 
-**Why**: GitHub Pages serves HTML with `cache-control: max-age=600`, and every deploy
-deletes the previous build's content-hashed CSS and JS. So for **ten minutes** after a
-deploy, a visitor can be holding HTML whose stylesheets no longer exist — a white page,
-no nav, the logo at its natural size, with a perfectly correct DOM. Reported from
-production four times (#438, #467, #476, #548).
-
-`scripts/retain-previous-assets.mjs` carries old assets forward so that visitor still
-resolves. It now chains across `RETAIN_GENERATIONS` (5) deploys — but before #548 it
-spanned exactly **one**, which is only enough at one deploy per cache window. Six PRs
-merged in 35 minutes, two of them 57 seconds apart, put real visitors two generations
-back and broke the site for them.
+**Why**: GitHub Pages serves HTML with `cache-control: max-age=600` and cannot be told
+otherwise, while every deploy deletes the previous build's content-hashed CSS and JS. A
+returning visitor can therefore hold HTML whose stylesheets no longer exist — a white
+page, no nav, the logo at its natural size, with a perfectly correct DOM. Reported from
+production **eight times** (#438, #467, #476, #548, #650, and three more through
+2026-08-15). The open ticket is **#635**; it stays open until the cause is gone.
+
+**The window is measured in DAYS, and getting that unit wrong is the recurring bug.**
+`scripts/retain-previous-assets.mjs` carries old assets forward, bounded by `RETAIN_DAYS`
+(14) in `deploy.yml`. It was previously bounded by a deploy count, and that was mis-sized
+twice — 5 (against the 10-minute cache window) and then 30 (against an assumed merge
+rate, which turned out to be 40 deploys in 6 days, so ~3.5 days). **Never restate this
+window in deploys**: converting requires a merge rate nobody measures, and
+`retain-previous-assets.test.js` now fails if `RETAIN_GENERATIONS` reappears (#751).
 
 **The trap is that docs-only PRs feel free.** `e2e.yml` has `paths-ignore` for
 `**/*.md`, `docs/**` and `.gitignore`, so markdown PRs skip the ~1-hour E2E mutex that
 paces everything else — and nothing else paces them. That is exactly how six merges
 landed in half an hour.
 
-**Now guarded by**: `scripts/check-stale-html.mjs` drives A → B → C and asserts a
-visitor holding A's HTML is still styled after **two** deploys, with a negative control
-that fails if one-generation retention ever stops breaking (i.e. if the harness has quietly
-stopped simulating a deploy). `scripts/__tests__/retain-previous-assets.test.js` asserts
-the chaining and the generation cap. Both run in CI — the first in the required
-`accessibility` check, the second via `pnpm test:scripts`.
-
-**Still worth pacing merges.** The guards make a burst survivable, not free: retention is
-capped at 5 generations, so more than five deploys inside one 10-minute window is still
-outside what anything protects.
+**Guarded by three things, which check different questions:**
+
+- `scripts/check-stale-html.mjs` (required `accessibility` check) drives A → B → C in a
+  real chromium and asserts a visitor holding A's HTML is still styled after two deploys,
+  with a negative control that fails if the harness stops simulating a deploy. It also
+  proves `StylesheetGuard` fires, stays inert on a healthy page, and re-arms after an hour
+  but not immediately (#752).
+- `scripts/ci/check-retained-assets.mjs` (post-deploy `smoke.yml`) reads the live ledger
+  and asserts both that every promised file is served **and that the window is still as
+  wide as `RETAIN_DAYS`**. The second assertion exists because the first was green on the
+  night production went unstyled for the eighth time.
+- `scripts/__tests__/*` via `pnpm test:scripts` for the chaining, the window and the unit.
+
+**The client-side backstop**: `StylesheetGuard` (in every page) detects a page whose
+same-origin stylesheets all have zero rules and re-fetches at a fresh URL. It is the only
+recovery that does not depend on a number being right — but it ships _inside_ the HTML,
+so a document cached before it existed has no guard at all.
+
+**Still worth pacing merges**, and still worth remembering that none of this is the fix.
+The fix is #635 — a CDN serving HTML `no-cache`, which GitHub Pages cannot be configured
+to do.
 
 ### NEVER bypass commit hooks (no `--no-verify`)
 

scripts/__tests__/check-retained-assets.test.js

@@ -16,9 +16,30 @@ function retainedEntries(extra = []) {
   ];
 }
 
-function runProbe(baseUrl) {
+/**
+ * An `ASSET_AGES.txt` body whose oldest entry is `spanDays` old (#751).
+ *
+ * The probe reads this ledger to judge whether the retention WINDOW is wide enough,
+ * which is a separate question from whether the files are reachable — and the one
+ * nothing asked on the night production went unstyled an eighth time.
+ */
+function agesFor(entries, spanDays) {
+  const now = Date.now();
+  return entries
+    .map((rel, i) => {
+      const age =
+        i === 0 ? spanDays : (spanDays * (entries.length - i)) / entries.length;
+      const when = new Date(now - age * 86400000).toISOString();
+      return `${i} ${when} ${rel.replace(/^\/+/, '')}`;
+    })
+    .join('\n');
+}
+
+function runProbe(baseUrl, env = {}) {
   return new Promise((resolve, reject) => {
-    const child = spawn(process.execPath, [SCRIPT, baseUrl]);
+    const child = spawn(process.execPath, [SCRIPT, baseUrl], {
+      env: { ...process.env, ...env },
+    });
     let stdout = '';
     let stderr = '';
     child.stdout.on('data', (chunk) => {
@@ -50,6 +71,10 @@ test('accepts a CDN-style 206 ranged GET when HEAD is unavailable', async (t) =>
       response.end(entries.join('\n'));
       return;
     }
+    if (request.url === '/_next/static/ASSET_AGES.txt') {
+      response.end(agesFor(entries, 20));
+      return;
+    }
     if (request.method === 'HEAD') {
       response.writeHead(405).end();
       return;
@@ -73,6 +98,10 @@ test('fails and names a missing retained stylesheet', async (t) => {
       response.end(entries.join('\n'));
       return;
     }
+    if (request.url === '/_next/static/ASSET_AGES.txt') {
+      response.end(agesFor(entries, 20));
+      return;
+    }
     if (request.url === missing) {
       response.writeHead(404).end();
       return;
@@ -88,3 +117,96 @@ test('fails and names a missing retained stylesheet', async (t) => {
   assert.match(output, /removed\.css/);
   assert.match(output, /STYLESHEETS/);
 });
+
+/**
+ * THE WINDOW ASSERTION (#751).
+ *
+ * Every check above asks whether retained files are REACHABLE. On 2026-08-15 all 13
+ * retained stylesheets were reachable and production was unstyled anyway, because
+ * the window they represented had shrunk to ~3.5 days while the config claimed a
+ * week. Reachability cannot see that; only these can.
+ *
+ * `RETENTION_RETIMED_AT` is backdated here because the floor is deliberately dormant
+ * during the ledger's first fortnight — without the override these would be testing
+ * the ramp, not the assertion.
+ */
+const PAST_RAMP = {
+  RETENTION_RETIMED_AT: '2026-01-01T00:00:00Z',
+  RETAIN_DAYS: '14',
+};
+
+const serveLedger = (entries, spanDays) => (request, response) => {
+  if (request.url === '/_next/static/ASSET_MANIFEST.txt') {
+    response.end(entries.join('\n'));
+    return;
+  }
+  if (request.url === '/_next/static/ASSET_AGES.txt') {
+    response.end(agesFor(entries, spanDays));
+    return;
+  }
+  response.writeHead(200).end();
+};
+
+test('fails when the retention window has collapsed below RETAIN_DAYS', async (t) => {
+  const entries = retainedEntries(['/_next/static/css/app.css']);
+  const server = await startServer(serveLedger(entries, 2));
+  t.after(() => server.close());
+
+  const result = await runProbe(server.baseUrl, PAST_RAMP);
+  const output = result.stdout + result.stderr;
+
+  assert.equal(result.code, 1, output);
+  assert.match(output, /covers only 2\.0 day\(s\)/);
+});
+
+test('passes when the window is at full width — the harness can reach success', async (t) => {
+  // Without this the test above passes just as well against a probe that fails on
+  // everything, which is the vacuous shape this repo keeps getting bitten by.
+  const entries = retainedEntries(['/_next/static/css/app.css']);
+  const server = await startServer(serveLedger(entries, 20));
+  t.after(() => server.close());
+
+  const result = await runProbe(server.baseUrl, PAST_RAMP);
+  const output = result.stdout + result.stderr;
+
+  assert.equal(result.code, 0, output);
+  assert.match(output, /full width/);
+});
+
+test('stays quiet during the ramp, when a narrow window is correct', async (t) => {
+  const entries = retainedEntries(['/_next/static/css/app.css']);
+  const server = await startServer(serveLedger(entries, 2));
+  t.after(() => server.close());
+
+  // Same 2-day ledger as the failing case; only the retime date differs.
+  const result = await runProbe(server.baseUrl, {
+    RETENTION_RETIMED_AT: new Date().toISOString(),
+    RETAIN_DAYS: '14',
+  });
+  const output = result.stdout + result.stderr;
+
+  assert.equal(result.code, 0, output);
+  assert.match(output, /still ramping/);
+});
+
+test('fails when the age ledger is missing entirely', async (t) => {
+  const entries = retainedEntries(['/_next/static/css/app.css']);
+  const server = await startServer((request, response) => {
+    if (request.url === '/_next/static/ASSET_MANIFEST.txt') {
+      response.end(entries.join('\n'));
+      return;
+    }
+    if (request.url === '/_next/static/ASSET_AGES.txt') {
+      response.writeHead(404).end();
+      return;
+    }
+    response.writeHead(200).end();
+  });
+  t.after(() => server.close());
+
+  const result = await runProbe(server.baseUrl, PAST_RAMP);
+  const output = result.stdout + result.stderr;
+
+  assert.equal(result.code, 1, output);
+  assert.match(output, /age ledger/i);
+});