Skip to content

Commit a21e714

Browse files
committed
refactor: move all.html into a Node.js generator
all.html, every module page's content on one page, is Node.js's, but html built it in, behind generateAllPage, and the streaming work spread its bookkeeping through generate.mjs. It's now a generator of @node-core/doc-kit, html-all, spliced between jsx-ast and html the way section-pages is between metadata and jsx-ast. It passes the pages on, adds all.html's metadata to the manifest, and right after the last module page adds all.html as a page made of theirs, in sidebar order. html only knows that an item with parts, the pages before it that it's made of, is compiled from their content and rendered first, one part at a time instead of building every part's elements before rendering any; generateAllPage goes away. Built with the Node.js preset and the html-all target, the Node.js API docs come out byte-identical, all.html included, at 1 and 4 threads. Assisted-by: Claude Opus 5.5 <noreply@anthropic.com> Assisted-by: Claude Sonnet 5.5 <noreply@anthropic.com>
1 parent 65d6aeb commit a21e714

25 files changed

Lines changed: 440 additions & 182 deletions

File tree

‎.changeset/html-all.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
---
2+
'@doc-kit/core': minor
3+
'@doc-kit/generator-react': minor
4+
'@node-core/doc-kit': minor
5+
---
6+
7+
`all.html` moves out of the `html` generator into a new Node.js generator, `html-all` (`@node-core/doc-kit`), and `html`'s `generateAllPage` option is removed. `html-all` is spliced between `jsx-ast` and `html`, and adds `all.html`, every module page's content on one page, as a page made of theirs. `html` renders such a page, given with `parts`, the pages it is made of, from their compiled content, one part at a time rather than building every part's elements first. Builds that want `all.html` add the `html-all` target.

‎.github/workflows/ci.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ jobs:
7777
path: node
7878

7979
- name: Build Node.js API docs
80-
run: node packages/cli/bin/cli.mjs generate -t web -i "./node/doc/api/assert.md" -o out --config-file ./e2e/doc-kit.config.mjs
80+
run: node packages/cli/bin/cli.mjs generate -t web -t html-all -i "./node/doc/api/assert.md" -o out --config-file ./e2e/doc-kit.config.mjs
8181

8282
- name: Install Playwright browsers
8383
run: pnpm exec playwright install --with-deps

‎docs/generators.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ Outputs consumed by the Node.js project's own build and release processes.
4848
| [`man-page`](./generators/man-page.md) | The `node.1` manual page. |
4949
| [`api-links`](./generators/api-links.md) | A map of API symbols to their source locations. |
5050
| [`addon-verify`](./generators/addon-verify.md) | Extracted addon code samples, arranged for testing. |
51+
| [`html-all`](./generators/html-all.md) | `all.html`, every module page on one page. |
5152

5253
## Pipeline stages
5354

‎packages/core/src/generators/index.mjs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ export const publicGenerators = {
1515
'json-simple': '@doc-kit/core/json-simple',
1616
'legacy-html': '@node-core/doc-kit-legacy/legacy-html',
1717
'legacy-html-all': '@node-core/doc-kit-legacy/legacy-html-all',
18+
'html-all': '@node-core/doc-kit/html-all',
1819
'man-page': '@node-core/doc-kit/man-page',
1920
'legacy-json': '@node-core/doc-kit-legacy/legacy-json',
2021
'legacy-json-all': '@node-core/doc-kit-legacy/legacy-json-all',

‎packages/node/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ npm install --save-dev @doc-kit/core @node-core/doc-kit
2020
| `man-page` | The `node.1` manual page, from the CLI options documented in `cli.md`. |
2121
| `api-links` | A mapping of API symbols to their source locations, powering source links on nodejs.org. |
2222
| `addon-verify` | Extracted C++ addon code samples from `addons.md`, arranged for compilation by the tests. |
23+
| `html-all` | `all.html`, every module page's content on one page, rendered by the `html` generator. |
2324

2425
## Usage
2526

‎packages/node/package.json‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
"./addon-verify": "./src/addon-verify/index.mjs",
1414
"./api-links": "./src/api-links/index.mjs",
1515
"./config": "./src/config/index.mjs",
16+
"./html-all": "./src/html-all/index.mjs",
1617
"./man-page": "./src/man-page/index.mjs",
1718
"./package.json": "./package.json"
1819
},
@@ -26,6 +27,7 @@
2627
],
2728
"dependencies": {
2829
"@doc-kit/core": "workspace:*",
30+
"@doc-kit/generator-react": "workspace:*",
2931
"estree-util-visit": "^2.0.0",
3032
"unist-util-visit": "^5.2.0"
3133
},
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# `html-all` Generator
2+
3+
The `html-all` generator adds `all.html`, every module page's content on one
4+
page, to the pages the [`html`](./html.md) generator renders. It builds nothing
5+
itself: it is spliced between `jsx-ast` and `html`, passes the pages on, adds
6+
`all.html`'s metadata to the manifest, and once every module page has passed,
7+
adds a page made of theirs. `html` assembles it from their compiled content,
8+
so no module is built twice.
9+
10+
The module pages are its parts, in the order the sidebar lists them; the
11+
index, section pages and the other synthetic pages are left out.
12+
13+
```sh
14+
npx @doc-kit/cli generate -t web -t html-all -i "doc/api/*.md" -o out
15+
```
16+
17+
## Configuring
18+
19+
It has no options of its own.
Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
import assert from 'node:assert/strict';
2+
import { describe, it } from 'node:test';
3+
4+
import { generate } from '../generate.mjs';
5+
6+
/**
7+
* A page as `jsx-ast` gives it.
8+
*
9+
* @param {string} api
10+
* @param {string} name
11+
* @param {object} [extra] - More of its metadata
12+
*/
13+
const createPage = (api, name, extra = {}) => ({
14+
data: { api, heading: { depth: 1, data: { name } }, ...extra },
15+
headings: [{ slug: api }],
16+
content: '<></>',
17+
});
18+
19+
/**
20+
* Reads every batch the generator yields.
21+
*
22+
* @param {AsyncIterable<Array<object>> | Array<object>} input
23+
*/
24+
const run = async input => {
25+
const batches = [];
26+
27+
for await (const batch of generate(input)) {
28+
batches.push(batch);
29+
}
30+
31+
return batches;
32+
};
33+
34+
describe('html-all', () => {
35+
const zlib = createPage('zlib', 'Zlib');
36+
const fs = createPage('fs', 'File system');
37+
const index = createPage('index', 'Index');
38+
const chunk = createPage('fs/promises', 'fs.promises', { chunk: {} });
39+
const notFound = createPage('404', 'Page not found', { synthetic: true });
40+
41+
it('adds all.html right after its last part, in sidebar order', async () => {
42+
async function* input() {
43+
yield [zlib];
44+
yield [{ manifest: [zlib.data, fs.data, index.data, notFound.data] }];
45+
yield [index, fs];
46+
yield [chunk, notFound];
47+
}
48+
49+
const batches = await run(input());
50+
51+
assert.deepEqual(batches[0], [zlib]);
52+
53+
const [{ manifest }] = batches[1];
54+
55+
assert.deepEqual(manifest.slice(0, -1), [
56+
zlib.data,
57+
fs.data,
58+
index.data,
59+
notFound.data,
60+
]);
61+
assert.equal(manifest.at(-1).api, 'all');
62+
63+
const [, , all] = batches[2];
64+
65+
assert.deepEqual(batches[2].slice(0, 2), [index, fs]);
66+
assert.equal(all.data.api, 'all');
67+
assert.deepEqual(all.parts, ['fs', 'zlib']);
68+
assert.deepEqual(all.headings, [{ slug: 'fs' }, { slug: 'zlib' }]);
69+
70+
assert.deepEqual(batches[3], [chunk, notFound]);
71+
});
72+
73+
it('adds all.html after every page given at once, as with one thread', async () => {
74+
const batches = await run([zlib, fs, { manifest: [zlib.data, fs.data] }]);
75+
76+
const pages = batches.flat();
77+
78+
assert.equal(pages.at(-1).data.api, 'all');
79+
assert.deepEqual(pages.at(-1).parts, ['fs', 'zlib']);
80+
});
81+
});
Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
'use strict';
2+
3+
import { toBatches } from '@doc-kit/core/utils/generators.mjs';
4+
import { headingSortFn } from '@doc-kit/generator-react/jsx-ast/utils/getSortedHeadNodes';
5+
6+
// The metadata of `all.html`
7+
const HEAD = {
8+
api: 'all',
9+
path: '/all',
10+
basename: 'all',
11+
synthetic: true,
12+
heading: {
13+
type: 'heading',
14+
depth: 1,
15+
children: [{ type: 'text', value: 'All' }],
16+
data: { name: 'All', text: 'All', slug: 'all' },
17+
},
18+
};
19+
20+
/**
21+
* Whether a page is part of `all.html`: the module pages, minus the index.
22+
* Chunk pages and the other synthetic pages are left out, as the module pages
23+
* carry their content.
24+
*
25+
* @param {import('@doc-kit/core/generators/metadata/types').MetadataEntry} data
26+
* @returns {boolean}
27+
*/
28+
const isPart = data => !data.synthetic && !data.chunk && data.api !== 'index';
29+
30+
/**
31+
* Builds `all.html` from its parts: their content one after the other, and
32+
* their tables of contents, in the order the sidebar lists them.
33+
*
34+
* @param {Array<Pick<import('@doc-kit/generator-react/jsx-ast/types').PageCode, 'data' | 'headings'>>} parts
35+
*/
36+
const buildAllPage = parts => {
37+
const pages = parts.toSorted((a, b) => headingSortFn(a.data, b.data));
38+
39+
return {
40+
data: HEAD,
41+
headings: pages.flatMap(({ headings }) => headings),
42+
parts: pages.map(({ data }) => data.api),
43+
};
44+
};
45+
46+
/**
47+
* Passes `jsx-ast`'s pages on, with `all.html`'s metadata in the manifest, and
48+
* `all.html` itself right after the last of its parts.
49+
*
50+
* @type {import('./types').Generator['generate']}
51+
*/
52+
export async function* generate(input) {
53+
const parts = [];
54+
55+
// The parts still to come, once every page's metadata is known
56+
let missing;
57+
let added = false;
58+
59+
for await (const batch of toBatches(input)) {
60+
const pages = [];
61+
62+
for (const item of batch) {
63+
if ('manifest' in item) {
64+
missing = new Set(item.manifest.filter(isPart).map(({ api }) => api));
65+
66+
for (const { data } of parts) {
67+
missing.delete(data.api);
68+
}
69+
70+
pages.push({ manifest: [...item.manifest, HEAD] });
71+
72+
continue;
73+
}
74+
75+
pages.push(item);
76+
77+
if (isPart(item.data)) {
78+
parts.push({ data: item.data, headings: item.headings });
79+
missing?.delete(item.data.api);
80+
}
81+
}
82+
83+
if (!added && missing?.size === 0) {
84+
pages.push(buildAllPage(parts));
85+
added = true;
86+
}
87+
88+
yield pages;
89+
}
90+
91+
// Without a manifest, once every page has passed
92+
if (!added) {
93+
yield [buildAllPage(parts)];
94+
}
95+
}
Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
'use strict';
2+
3+
import { generate } from './generate.mjs';
4+
5+
/**
6+
* Adds `all.html`, every module page's content on one page, to the pages the
7+
* `html` generator renders. It builds nothing itself: it passes `jsx-ast`'s
8+
* pages on to `html`, and, once every module page has passed, a page made of
9+
* theirs, which `html` assembles from their compiled content.
10+
*
11+
* @type {import('./types').Generator}
12+
*/
13+
export default {
14+
name: 'html-all',
15+
16+
description:
17+
'Adds all.html, every module page on one page, to the html generator',
18+
19+
dependsOn: '@doc-kit/generator-react/jsx-ast',
20+
21+
streamsInput: true,
22+
23+
dependent: '@doc-kit/generator-react/html',
24+
25+
generate,
26+
};

0 commit comments

Comments
 (0)