M1: the free probe, and a language that is part of a page - #1
Open
gordon2 wants to merge 174 commits into
Open
Conversation
The committed language map was wrong about 21 pages. Every over-long section
was followed by an exactly-matching under-long one, which is the signature of a
boundary recorded in the wrong place rather than a document that is unusual:
FI 233-256 / DA 257-264 -> FI 233-248 / DA 249-264
SR 361-380 / LT 381-392 -> SR 361-376 / LT 377-392
LV 393-412 / SK 413-424 -> LV 393-408 / SK 409-424
TR 441-461 / VI 462-472 -> TR 441-456 / VI 457-472
JA 539-560 -> JA 539-559
Each correction restores both sections to 16 pages, this manual's standard
section length. Three independent methods agree, which is why this is a
correction and not a preference:
* The manual prints its own language code on every content page. It reads FI
through 248 and DA from 249.
* Orthography, with no detector involved: pages 233-248 carry 1252 ae/oe
umlauts and zero aesc/oslash/aring, pages 249-264 exactly the reverse.
Latvian macrons appear on 393-408 and never after; Slovak ae and
o-circumflex appear from 409.
* lingua-go agrees on all four, and disagreed with the committed map on
precisely these 21 pages and no others.
The manifest was already internally inconsistent: DA recorded printed_page 243,
and printed folio 243 is on PDF page 249 — the value it stored was right and
its pdf_start was wrong. Same for LT, SK and VI.
Page 560 is a back cover, an English colophon with no page tag, so content ends
at 559 and Japanese is 21 pages.
Also corrected in the manifest and in ingest.md, where they were measurably
wrong rather than merely stale:
* median_chars_per_page 2241 was the median in BYTES; in characters it is
1693. Half this document is Cyrillic, Greek, Hebrew, Arabic or CJK, where
bytes run a third higher, so the distinction decides the text-layer
threshold.
* index_pages is 2-4, not 2-3. Page 4 is a third contents page.
* The printed-to-PDF offset does not drift +6 then +8. It is a constant +6
across all 34 sections. What drifts is what the index CLAIMS, on 10 of 34
sections, which is a different failure and the reason a claimed start is a
hypothesis rather than a boundary.
* The household scope is 48 of 560 pages, 8.6%, not 9.6%.
* CZ's real folio is 307, not 305.
Recorded in the manifest: the per-page language tag, its guards, and the two
sections detection cannot label (Uzbek, unsupported by lingua-go at all, and
Latin-script Serbian, read as Croatian on every page).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four signals can say what language a page is in, and the choice between them is
not obvious enough to leave undocumented. Measured on the 560-page fixture:
printed page tag free, exact when present 553/553 pages
printed index free labels and titles, page claims 1-2 off
Unicode script free 151/554 pages, 6 languages outright
statistical a dependency 93.7%
The page tag is the cheapest and the best: it arrives as the first line of the
pdftotext output the text probe already produces, and it labels the two sections
detection cannot — Uzbek, which lingua-go does not support at all, and
Latin-script Serbian, misread as Croatian on all 16 of its pages. It is not
universal across manuals, so it is a shortcut rather than a replacement.
The lingua-go numbers are why the detector decision is deferred rather than
made:
* WithPreloadedLanguageModels() over 75 languages costs 2154 MB resident and
buys nothing — low-accuracy mode scores an identical 93.7%, because manual
pages average ~1700 characters and lingua's advantage is on short strings.
* The binary goes 11.7 MB -> 129.5 MB and cannot be pruned. Linking only
German, English and Ukrainian still produces 129.5 MB, because the models
are go:embed'ed as one directory.
* Accuracy tops out around 94% and the residue is systematic, not random.
Deferring it on a sample of one document would be the wrong call in the other
direction, so the page states what to measure once a real library exists. The
pipeline is built on the three free signals; SourceDetector exists as a constant
so adding one is not a rewrite.
CLAUDE.md records the conventions that have actually caused bugs here — count
runes not bytes, strip Unicode format characters before matching text, handlers
must be idempotent — alongside the standing expectation to measure rather than
estimate and to verify where the user will see it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Stages 0-2 of the funnel in docs/design/ingest.md. pdfinfo for the metadata,
one pdftotext invocation for the text, then the language map from several
signals reconciled against each other. Nothing here calls a model, touches the
network, or costs anything: on the measured fixture it reduces a 560-page,
34-language manual to the 48 pages a de/uk/en household reads, in 1.9 seconds.
The reconciliation rule generalises ingest.md past two signals: prefer the
cheapest signal present, corroborate with the next, record conflicts rather than
resolving them silently, and leave a page nobody could name unnamed.
Five things here exist because testing against a real document found them, and
every one is invisible on a synthetic PDF:
* Right-to-left pages wrap Latin furniture in bidirectional control marks, so
the tab reading "HE" is really RLE LRE H E PDF PDF. Without stripping format
characters the entire Hebrew and Arabic sections went unlabelled.
* A contents page's trailing number is an index entry's page reference, not a
folio. Page 2 ends with "194", so Arabic's claimed start of 194 resolved
onto page 2 and produced a one-page Arabic section at the front.
* A contents page adjacent to the first section it lists is absorbed into it;
the run-length guard only helps when they are separated.
* The index's final entry claims every remaining page, which swallowed an
English back cover into the Japanese section. Script and language must be
checked against each other in both directions.
* A section split across a full-page illustration still totalled the right
number of pages, so a totals-only assertion missed it.
Grouping distinguishes "a vaguer signal filled a gap" from "the document named
two variants": a script signal cannot express a region, so it generalises into
its neighbour, but a page tag naming PT on one page and BR on another is two
sections. Merging them would scope a household onto both and charge it to
translate the wrong sixteen pages.
internal/testpdf generates small valid PDFs in memory. No PDF may be committed
here — CI rejects every .pdf, because a committed document is either someone's
copyrighted manual or someone's private paperwork — but the pipeline can only be
tested honestly against a real one read by real poppler.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Migration 00002 plus the service over it. Two schema decisions are load-bearing rather than stylistic: Derived tables use composite natural keys, not ULIDs. A job handler can run twice — a worker may be killed after doing its work but before recording success — so the probe must be able to write its results again without duplicating them. A natural key makes "run it again" an upsert; with surrogate keys the second run would insert a parallel set of 560 page rows and the reconciliation would silently double. doc_langs stores every signal's view, not just the reconciled one. "This manual also contains FR, IT, ES..." must be answerable without re-probing, and a conflict has to stay inspectable after the fact. pdf_start = 0 means "named a language but could not place it", which is a real state and not a defect to reject. A printed index routinely claims a page that does not exist, or one whose script makes the claim impossible — this manual lists Czech at a page that is Arabic. The claim is still evidence that the contents table is wrong. An earlier CHECK of pdf_start >= 1 failed the entire probe on the real document, and keying on the page alone would have collapsed every unplaceable claim into one row, so the code is part of the primary key. Serial numbers and purchase prices are deliberately absent. They are the highest-harm fields manualbox will hold and must be encrypted with a key kept outside the data directory; adding them in the clear now would mean migrating real user data later. They land with the keyring. See docs/design/privacy.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The vertical slice: create a device, upload a PDF, and the server reads it
locally and reports what it is holding, then stops and waits. On the real
560-page manual it says
This manual contains 34 languages across 560 pages. Yours are 3 of them —
48 pages, 9% of the document.
and lists the other 31 so importing one later is a button rather than a
re-upload. Verified in a browser, end to end, against that document.
No conversion happens. The Import button is disabled and says why, because
conversion does not exist yet and offering a button that does nothing is worse
than saying so. Every probed document stops at awaiting_scope.
The gate shows no cost figure. A token count needs a specific model's tokeniser
and a currency figure needs a billing mode — metered bills money, a subscription
draws down a window, a local model costs nothing — so with no provider
configured there is no honest number and it says that instead of guessing one.
Characters, which are measured and free, are shown. See docs/design/providers.md.
Security, from review of this code:
* The client's Content-Type was stored verbatim and echoed back with
Content-Disposition: inline. Uploading an .html file — as anyone might, from
a vendor site — served its script from this instance's own origin, next to
the session cookie, where checkOrigin cannot see it. The media type is now
sniffed from the stored bytes, only PDFs and raster images may render
inline, everything else downloads, and nosniff is set on every response.
text/plain is excluded from the inline list: Go's sniffer returns it for any
unrecognised text including SVG, and one response header should not be all
that stands between stored bytes and script execution.
* Absolute blob paths reached documents.last_error, the API and the log. The
data directory usually sits under a home directory, so that is an OS
username in the artefact users paste into public issues. Errors now carry
the digest.
* The poppler subprocesses had no timeout. The job context is cancelled only
at shutdown while the lease renews for as long as the handler runs, so a
hung pdftotext held a worker for ever and two such documents stopped all
ingest permanently. 30 s and 5 min, plus a 64 MB output cap against a PDF
whose compressed streams expand enormously.
* filepath.Base does not strip a Windows path on Linux, so an uploaded
C:\Users\alice\... arrived intact, storing a stranger's username.
* The route-guard test promised that a new unguarded route shows up as a
failure, but eleven new routes were never added to it. They were all
correctly guarded; now it is enforced.
store.Path hands a real filesystem path to poppler, which needs a file rather
than a reader. Safe because blobs are immutable and mode 0400, and the digest is
validated — now covered by the traversal test that the #nosec justification
already named.
jobs.RunOnce lets a test drive the queue deterministically through the real
claim-and-execute path, which is what makes the idempotency and permanent-
failure tests assert on a finished job rather than a timeout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pdftotext cannot report font weight, and weight is the discriminator. Measured
on the fixture's English section, by character share:
21pt bold 0.9% headings ("Safety Information")
17pt regular 14.6% BODY ("• This product should not be used by...")
15pt bold 2.1% headings ("Usage Restrictions")
11pt bold 7.0% headings ("Spot Cleaning Button")
11pt regular 57.7% body
9pt regular 17.1% notes
A "larger than body means heading" rule promotes that 17pt safety copy — 62 runs
of prose — into headings. Weight separates them cleanly, and only pdftohtml's
XML output carries it, along with font size and family.
Adding it to All() is what makes doctor and the instance endpoint report it: a
tool the pipeline needs but never mentions is one the user cannot be told to
install. doctor now lists 6 of 6.
No Dockerfile change needed — verified rather than assumed by installing
poppler-utils in alpine:3.21, the image's own base, where pdftohtml is present
at /usr/bin/pdftohtml.
Nothing calls it yet. Heading classification is the next deliverable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Languages that share a script do not share an alphabet. Unicode script analysis narrows a Cyrillic page to seven candidates and stops; counting the letters only some of those seven can write costs nothing and finishes the job. The case that forced it: a manual with Russian, Ukrainian and Kazakh side by side, one per column. Per column, on 18 of the 19 such pages (the exception being a page of contact addresses): column uk marks ru marks kk marks verdict left 0 40 0 Russian middle 83 0 0 Ukrainian right 78 111 143 Kazakh A maximum over those counts reads the right column wrong. Kazakh's alphabet CONTAINS the i it shares with Ukrainian and the yeru it shares with Russian, so overlap is the normal case rather than a conflict. Two questions decide instead: can this language write everything observed, and does the text exercise this language. Russian cannot account for 67% of the right column and Ukrainian cannot account for 76%, so both are excluded by what they cannot write. On the LEFT column Kazakh can account for everything, since Russian's alphabet is a subset of its own — what settles that one is that none of Kazakh's own nine letters appear. Either question alone gets one of the three columns wrong. Cost: +1,536 bytes of binary and 100 microseconds per page, against lingua-go's +118 MB and 4.0 s. On 31 paragraphs, one per language: 25 named, 6 declined, 0 named wrongly. Blind spots are part of the contract and are tested, not merely documented. Danish and Norwegian share their whole repertoire; so do Serbian, Croatian and Bosnian in Latin script; English, Indonesian and Malay have no distinctive letters at all. Those return a tie or nothing rather than a guess. Czech and Slovak, which I had assumed were a blind spot, separate cleanly — Czech cannot write l-caron or o-circumflex — so this signal reaches a case a trigram detector does not. Call it per column, never per whole page: a language whose alphabet contains most of its neighbours' will absorb a multi-column page. Measured margin on a real page is 7.25% foreign against a 5% threshold, which holds, but narrowly. Nothing is wired into reconciliation yet; that needs column geometry first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A language can be a column rather than a span of pages, so the pipeline has to
find columns before it can find languages. Correct on all eight pages of a real
manual that were checked against their rendered images.
Three approaches were tried. The two that failed are recorded because each is a
trap that looks right until it is measured:
WHITESPACE PROJECTION is binary. Project each run's x-extent, split at gaps, and
one heading set across the measure welds two columns together for ever. It got 63
of 68 pages right and merged the rest.
LEFT-ALIGNMENT PEAKS over-split, because alignment is local. A three-column parts
list yields six peaks, each column having a hanging indent for its numbered
markers; another page yields a spurious peak 162px into a column from a nested
list. No fixed merge distance separates a 30px hanging indent from a 162px
sub-indent while keeping two real columns 280px apart.
COUNTING CROSSINGS works because it is page-wide, which is what a column boundary
is. For each x, how many runs cross it; a gutter is a band FEW runs cross, not
none. One spanning heading no longer welds anything: the pages that failed before
have exactly one and two spanning runs.
Two filters were needed before any of it worked, and both were got wrong first:
* Production artifacts. An InDesign slug and an export timestamp, 261
occurrences each across 67 of 68 pages, several sitting in gutters. Filtering
by repetition is wrong in both directions — across pages it also matches the
printed UA and PL language tags, and within a page it misses the two pages
carrying only two copies while deleting 742 of 769 genuine runs on a page that
legitimately prints a company name a dozen times. The discriminator is height:
522 runs at 2-6px against a body median of 17, being leftovers scaled down
with placed artwork.
* Off-page runs. One page parks 218 runs at negative coordinates, invisible in
print and lying across two gutters. This is the only filter that changes a
column count.
Figure callouts are excluded by how much text a candidate holds: real columns
carry 1,116-3,058 characters, an exploded diagram's callouts carry 12 and 24, and
a parts list with short lines sits between at 1,716 — so a character count
separates them where a median line length would not.
Every parameter is a named constant with the measurement that justifies it, and a
sweep of the two most sensitive shows the chosen point sitting interior on both
axes rather than balanced on an edge.
Column widths vary within one document — 262px on three-column spreads, 403px on
wide two-column ones — so nothing assumes a fixed width, count or pitch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The sectioned fixture cannot express a manual whose languages share a page, so a pipeline built against it alone found 1 of this document's 5 languages and left 56 of 68 pages unlabelled. This is the counter-example. The PDF is not committed: 9 MB of third-party copyright, fetched on demand like the other one. Section gains PageFact beside it, because the unit of truth differs by document. A sectioned manual records contiguous spans; a column manual records each page's columns, since no language has a span to record. PROVENANCE IS PER PAGE, and it is the point rather than bookkeeping. Eight pages are marked "image": a human compared them against their rendered page and confirmed the column count. Those are ground truth and may be asserted against. The other sixty are marked "detector": produced by DetectColumns, recording the current reading rather than established truth. Holding a detector to its own output is circular, and the test enforces that both kinds exist so the file cannot quietly become self-confirming. That distinction exists because the first version of this file did not have it. It was generated by an ad-hoc script splitting columns at gaps wider than 90px, and it was wrong on 4 of the 8 pages later checked by eye — the real gutters here are 9 to 17px. It was offered as an answer key while being wrong. Languages come from the character-repertoire signal, which is independent of the column geometry, so a column's language is not evidence produced by the code that found the column. 165 of 169 columns are named. The gaps are deliberate: one is a narrow table cell of German read as Finnish, demoted to unestablished with the reason recorded, because a wrong entry in an answer key is worse than a missing one. known_limitations records what the fixture cannot settle, so its silence is not mistaken for coverage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two real manuals disagree about something the pipeline treated as fixed: whether a
language occupies pages or part of a page. Recording it because the answer took
three wrong measurements to reach, and each wrong turn is a trap the next person
will meet.
The design conclusion is that there is no Layout interface. An earlier draft
proposed one, with an implementation per arrangement chosen per document by a
scored Detect, and it was wrong twice over — a document contains several
arrangements, so a per-file choice is confidently wrong on every page it does not
fit, and the interface bundled two things that fail separately. What replaces it
is a geometry pass that knows nothing about language, and an assignment rule above
it. The payoff is already visible: the geometry pass shipped and was verified
before the assignment question was answered.
Also corrected here, all of them my own published errors:
* The column distribution was 11/16/40/1. It is 3/31/28/5 plus one blank page.
The first figure came from a script that could not see a 17px gutter.
* "The gutter simply is not there" on the three-column parts list. It is there:
9px, and the threshold was too wide to see it.
* A claim that a realistic multi-language page fools the repertoire signal. It
does not; that came from a constructed sample repeating one thin sentence.
The index discriminator keeps its measurement and gains its provenance: those 87
entries came from an ad-hoc script, not the shipping parser, which requires a
two-letter code and could not produce them. Real signal, unreproduced measurement,
and now labelled as such.
Failing honestly is stated as a rule with teeth: layout classification must never
veto a stronger signal already present. An earlier draft made "unclassified" label
nothing, which would have taken a manual that works today to zero languages. The
fallback is current behaviour with a flag, and the flag drives a route — keep the
original, process locally, or ask a model — rather than a refusal. Whether a model
is actually better on hard documents is untested and measurable, so it is not
claimed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One manual marks its five languages D, PL, RUS, UA and KAZ. An exactly-two-letter matcher reads two of them, so the page-tag signal — the cheapest and most accurate one there is — produced zero runs on that document. Now one to three letters, with an optional region. Widening the shape alone was not enough and briefly made things worse, in two ways worth recording because both look safe until measured: A THREE-LETTER SHAPE IS NOT A THREE-LETTER CODE. golang.org/x/text accepts "one", "two", "the", "and", "for" and "abc" as languages — real ISO 639-3 codes for languages no appliance manual is printed in. Letting a three-letter token fall through to the parser turned the first word of a page into a language tag, and "one" was duly read as a code. So membership is by length: two letters go to the parser, which knows the small closed set of ISO 639-1; one and three letters must appear in the table of codes manuals actually print. That table is ISO 639-2/B, the vernacular form — GER rather than DEU — because that is what gets printed. A SINGLE LETTER CANNOT BE TRUSTED FROM POSITION. Page 511 of the 34-language manual opens with a figure label "F" and carries its real ZH-HK tag on the next line. Reading the F as French split the Cantonese section in two, which the fixture caught. A single letter is now believed only where the document's own contents table lists that code — which is exactly the case on the manual that prints D for German, and is never the case for a stray figure label. Both guards are the same principle the rest of this package runs on: a signal may be present without being trustworthy, and corroboration decides which. Also: three-letter labels now parse as their own index entry rather than merely stopping the lookahead, so the section they name is no longer lost. The test that asserted otherwise encoded the old limitation and now asserts the better behaviour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A page of the column-layout manual holds up to three languages, so every question about naming one has to be asked per column: a printed tag belongs to one of them, and one column's alphabet says nothing about its neighbour. Two signals combine, and the measurement is the argument for having both. Across the 169 columns of that manual: named by the printed tag 79 named by the alphabet 87 unnamed 3 conflicting 2 pages where BOTH were needed 36 of 68 Neither signal covers half the document, and on more than half the pages each one names columns the other cannot. On page 2 German and Polish come from printed tags while Russian comes from its alphabet; on pages 13 and 41 it is the reverse. Read per page instead of per column, all of them collapse to one answer or none. Verified against the eight pages a human checked against their rendered images: 7 of 7 multi-column pages fully named, every column correct. Where the two disagree the printed tag wins — it is the document asserting its own language — but the conflict is recorded rather than resolved silently, naming what the alphabet read instead, because one of them is wrong and the reader is better placed to say which. A single-letter tag is taken only where the document's own contents table lists that code. "D" is German on a manual whose index lists D, and a figure label everywhere else; that distinction already cost a section boundary once. The tag is searched over a column's first few runs rather than its first, because a right-to-left column prints its heading before the tab in reading order. Nothing is stored yet. The schema has one language per page and its CHECK does not allow a repertoire source, so persisting this needs a migration and a re-keyed reconciliation — deliberately a separate piece of work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The next change alters tables the first slice already committed, and self-hosted means a bad migration is other people's data. So the decisions go down first, from a survey of what actually breaks rather than from memory. The concrete blocker: doc_langs is keyed (document_id, source, code, pdf_start), so two German columns on one page collide — same page, same code, same source, and nothing to tell them apart. Keying on geometry instead, (document_id, source, page, x0), separates them and keeps the natural-key upsert that makes re-probing idempotent. A surrogate key would break that: a second probe would insert a parallel set rather than converging on the same rows. A new doc_regions table rather than widening doc_pages, because they record different things — a page does have one dominant script, one printed folio, one tag position, and those stay per page. A whole-page region carries no box, so a sectioned manual stores exactly what it stores today and page-only readers keep working. Characters replace pages as the unit of size. "48 of 560 pages" was always a proxy and it stops meaning anything when a page holds three languages. That needs text slicing by rectangle, which does not exist anywhere yet, so it is part of this deliverable rather than a follow-on: Scope.Chars is wrong the moment regions land without it. repertoire joins the source CHECK lists in an append-only 00003. 00002 is shipped; editing it would diverge from any database already created from it. Four things are left unsolved on purpose and are written down as such, so they are not mistaken for oversights: regions do not compose across pages, language-neutral content has no home, a table cell cannot be told from a text column, and interleaved paragraphs would make a region a paragraph annotation instead of a layout partition. The last one is a stated stop condition — if a third manual does it, this design is the wrong shape rather than an incomplete one. Acceptance is not "the migration applies": the Thomas manual's five languages must read back column for column against its eight human-verified pages, and the Dreame manual's 34 sections must be unchanged. Improving the second by altering the first means something broke. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The README told people they could not add devices or manuals. On this branch they can: create a device, upload a manual, and be told what it contains and in which languages before anything is converted or sent anywhere. It stops at the gate, and the note now says so rather than overstating in either direction — the previous text would have read as false the moment this merged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DetectColumns and ColumnLanguages have existed for two commits and nothing could
feed them: both were only ever called with coordinates typed into a test, and the
fixture's own per-page entries came from a script that no longer exists. So the
column work was unverified against what poppler actually reports, and the
acceptance criterion in docs/design/regions.md — the five languages read back
column for column — had no path from a PDF to a column at all.
ExtractRuns closes that. One pdftohtml -xml invocation per document, measured:
560-page, 15 MB manual 1.79 s 3.8 MB of XML 34,413 runs
68-page, 9 MB manual 3.18 s 920 KB of XML 7,493 runs
the same order as the pdftotext pass beside it, so the free stages stay free.
Coordinates come back at 1.5x the PDF's points — 108 dpi against 72 — which is
why a pdftoppm -r 108 raster matches this space 1:1 and a detected box can be
drawn on the page and looked at. Both fixtures confirm it: 918/612.283 and
892/595.276.
The reader unmarshals with a custom UnmarshalXML, and the reason is measured
rather than assumed. Poppler wraps a styled run's text in <b> or <i>, 355 runs of
the column manual are wrapped, every one of them whole — so Go's `,chardata`
returns nothing at all for them, and the printed language tabs D, PL and UA are
among them. Read both ways over that document's 169 columns:
naive correct
columns named 166 167
named by printed tag 0 53
named by its alphabet 166 114
tag/alphabet conflicts 0 1
The first guess — that this loses the signal the pipeline is built on — was wrong,
and the code now says what is true instead. The count barely moves, because this
manual's five languages have distinguishable alphabets. What collapses is
attribution, and the one place the two signals disagree stops being visible. A
manual whose languages share an alphabet is precisely the case the printed tag
outranks everything for, and there nothing would be left.
Verified against the eight pages a human compared with their renders: column
counts, spanning runs, every column's x-range and run count, and 16 of 16 columns
whose language the manifest names. That is the first non-circular check of either
detector.
testpdf grows a Column, so a two-column page can be generated in memory and read
back through real poppler — that end-to-end check needs no network and runs in the
default suite. It found the 918x1188 box and both columns at the offsets asked for.
Two things this surfaced, neither introduced here and neither fixed here:
IndexRuns cannot parse the column manual's contents page. It yields the vocabulary
[FAX GA NDE UA VIA Z], of which only UA is a language, so columnTag rejects every
German column's printed "D" for want of corroboration and falls back to its
alphabet. Supplying the true vocabulary by hand raises tag naming from 53 to 79 —
which is the figure the commit introducing ColumnLanguages recorded, measured with
a hand-supplied list rather than through the assembled pipeline. The totals hide
it, so a test now pins the split.
The sectioned manual is not single-column, which a first version of the new test
asserted and which is wrong: 199 of its 560 pages read as three columns and 148 as
one. Pages 20 and 100 rendered at 108 dpi settle it — both are two side-by-side
troubleshooting tables, and the regions returned are their cells, correctly placed.
On page 20 only the two wide answer cells come back, the narrow ones falling below
minColumnRuns, which is that guard working. The assumption was wrong and the code
was right; the distribution is now recorded in the manifest rather than assumed,
and the test claims nothing about it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`repertoire` has existed in Go as doc.SourceRepertoire since the signal was written, but 00002's CHECK lists name the other five and omit it. Storing anything from that signal would fail a CHECK at runtime, in a background job, against a user's own database. This commit is only that correction. It is deliberately separate from the doc_regions table that motivated it (docs/design/regions.md): a schema change to other people's data should be reviewable and revertible on its own, and nothing writes 'repertoire' to these two columns yet. No Go behaviour changes. SQLite cannot alter a CHECK — the constraint is part of the stored CREATE TABLE text — so 00003 rebuilds both tables: create with the widened list, copy with named columns on both sides, drop, rename, recreate the three indexes. 00002 is never edited; a database already created from it would diverge. Both tables are foreign-key leaves, so the drop cannot orphan a child row and their parent, documents, is untouched. What was actually verified, not assumed: - The rebuild runs inside goose's default transaction with the connection's _pragma foreign_keys(1). Measured by temporarily appending a failing statement to the Up block: UpTo(3) returned "partial migration error", and afterwards doc_pages still carried 00002's DDL, doc_pages_lang_idx was still present, no scratch table survived, and the version was still 2. So no NO TRANSACTION escape hatch is needed and none was used. - `make sqlc` regenerates internal/db/gen byte-identically. sqlc reads internal/db/migrations as schema truth and survives create/copy/drop/rename; only CHECK text differs, so no generated type moves. - `make check` green: go test -race -shuffle=on, golangci-lint 0 issues, tsc --noEmit clean. The test brings a real file-backed database to version 2, writes rows covering every column — NULL printed_folio, NULL printed_page, a page with no text layer, and the pdf_start = 0 "named a language but could not place it" row that 00002 documents at length — then migrates to 3 and compares every column of every row as text, so a lost NULL reads differently from a lost value. It also asserts the constraint was widened rather than dropped, that 00002's other CHECKs, STRICT and ON DELETE CASCADE all survived, and that Down returns to the narrower shape with the rows intact. Then the fix was reverted three ways to confirm the test is not worthless: - narrowing 00003's lists back to 00002's: caught, the 'repertoire' insert is rejected - dropping the two CHECK clauses entirely: caught, the bogus values are accepted where the test requires rejection - omitting printed_folio from the copy: caught, row 0's printed_folio reads <NULL> where 3 was written Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The change docs/design/regions.md contracted for, in the doc package only: nothing
is persisted yet. A page of the column manual holds up to three languages, so the
unit stops being the page and becomes a region — the whole page where a manual runs
its languages in sequence, a box where it runs them in parallel.
The one decision the contract left open was when a page divides, and it needed both
manuals to settle rather than either alone:
- The column manual sets two columns of one language on pages 6 to 10 and three
on 52 to 56. Its own manifest says column count identifies nothing.
- The sectioned manual reads as three columns on 199 of its 560 pages and four on
71, and every one is a side-by-side troubleshooting table. Rendered at 108 dpi
and checked by eye on pages 20 and 100.
So a page divides on LANGUAGE, never on geometry. Dividing on geometry stored four
regions for a single-language page across hundreds of pages of a manual with no
parallel columns at all.
Precedence, and each branch is measured:
1. A page the per-page signals named is one whole-page region in that language.
On the sectioned manual the printed tab is right on all 553 content pages, and
letting a column's alphabet overturn it split 31 pages and contradicted the tab
on 46 regions — "de" read as Finnish, Spanish and Portuguese, every one a short
table cell. A disagreeing column now records a conflict and changes nothing.
2. Otherwise the columns divide the page if they name more than one language. On
the column manual the per-page signals name NOTHING on any of its eight
verified pages, which is why the columns are trusted there and not here.
3. Otherwise the columns' single language names the page, unless it is a contents
table, whose own letters are no guide: the sectioned manual's pages 2 to 5 read
as Swedish and Turkish and the column manual's address page as Turkish.
4. Otherwise no language, which is reportable rather than a failure.
A first attempt made rule 1 depend on how much evidence the alphabet had, on the
theory that the bad readings were starved of it. Measured against 685 labelled
columns of the sectioned manual, that is false: the signal is 93% accurate and its
mistakes are spread across every mark count including one with 118. There is no
threshold there, so there is no threshold here.
Rule 1 defers to the per-page answer only where that answer names a language
anything recognises. It need not: BCP-47 constrains a subtag's shape and not its
meaning, so FAX parses as "fax", TEL as "te", NDE as "nd". Not hypothetical — the
column manual prints FAX on its service-address page, the index parser reads that
page as a contents table, and reconciliation labelled two pages "fax", overriding
columns that read correctly as German and Polish. KnownLanguage now gates what may
outrank other evidence, while leaving what may be stored alone: an unrecognised
code a manual really prints is still kept, because that is information.
Characters replace pages as the unit of size, which is the part Scope.Chars was
already wrong about. Text is sliced by rectangle over the runs the detector counts
as text — excluding the 522 sub-legible production slugs and the 218 runs parked
above one page's top edge, which overstate that page by half. Measured effect on
the column manual: a German-reading household is charged 44,376 characters rather
than the 233,849 of every language sharing those pages. 19%, where before regions
one language of five cost the same as all five.
The two tools disagree slightly about how many characters a page holds — 3.3% and
2.5% on the two documents' totals, 1 to 2% on a median page, up to 51% where a text
layer parks runs off the page — so regions count with one tool throughout rather
than mixing them, and the fallback path where pdftohtml is absent says which
measurement it is.
Languages() now summarises regions where there are regions. Not a preference:
summarised from per-page runs alone the column manual reports no languages at all
while plainly containing five. On the sectioned manual the two agree exactly, which
is asserted rather than assumed — all 34 sections, their page counts, their first
and last pages, and one contiguous span each.
Acceptance, both halves: the column manual's five languages read back across its
parallel columns, with the eight human-verified pages matching column for column;
the sectioned manual produces exactly 560 whole-page regions over 560 pages, none
boxed, 34 languages, and its language map is untouched.
Both load-bearing decisions were reverted to confirm the tests catch them. Splitting
on column count instead of language fails the two-columns-of-one-language test;
accepting any parseable tag fails the FAX test and puts "fax" back among the
document's languages.
Still not persisted, and deliberately: the migration and SaveProbe are the next
commit, so a schema change to other people's data is reviewed on its own.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The design docs carry the reasoning and the measurement, so three findings from building the region model belong in them rather than only in a commit message. Two contradicted the attempt that produced them. layouts.md said a Dreame page holds one language, which is true, and it was read as also meaning one column, which is false. Measured over all 560 pages: 199 read as three columns, 136 as two, 71 as four, 148 as one. Pages 20 and 100 rendered at 108 dpi settle it — both are two side-by-side troubleshooting tables and the regions are their cells. So column count is not language count in EITHER direction: one Thomas page holds several languages, one Dreame language is set across several table cells. Any rule keyed on column count is wrong on one of the two documents. That also answers what layouts.md called its main open question. A table cell cannot be told from a text column by geometry, and now nothing asks: a page divides only where its columns name more than one language, so a same-language table is one region. Thomas pages 57-61 and 406 Dreame pages are disposed of together. A table whose cells are in different languages would still divide and be wrong to, and is recorded as unsolved rather than guarded against, since the guard would be written against an imagined document. language-detection.md gains the two measurements that changed decisions: The printed-index parser cannot read the Thomas contents page. It yields the vocabulary [FAX GA NDE UA VIA Z], of which only UA is a language. So "D" is not in the document's own vocabulary, every German column's printed tab is rejected for want of corroboration, and tag-named columns fall from 79 of 169 to 53. The total named barely moves, which is why nothing failed loudly. Worse, FAX parses as a language tag and became two pages' reconciled language over columns that read correctly. The repertoire signal is 93% accurate, not the 25-of-31 the hermetic paragraphs suggested — measured over 685 columns of the L40 against its printed tab, which is right on all 553 content pages. And its errors do not correlate with how much evidence it had: accuracy is flat at every cut from 1 to 50 distinctive characters and one wrong naming carries 118. A minimum-evidence threshold was designed against that measurement and abandoned by it, which is worth recording precisely because it is the intuitive fix. regions.md gains a section on what building it settled: divide on language rather than geometry, let the per-page answer outrank a column's where it exists, require a page-level answer to name a real language before it may outrank anything, and count characters with one tool rather than two. Plus the stop condition it now has — a manual that both prints a whole-page tab and sets parallel columns of different languages would be called one language, and neither fixture is that document. One correction to the earlier sketch: a whole-page region's box is not absent but spans zero to the page width, so a reader clipping to the box gets the whole page instead of every caller carrying a null check. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`doctor` told the user this tool exists to "read font size and weight, which is how headings are found" — the speculative future use it was added for. It is now the tool that reads where text sits on the page, which is what a column is, and on a manual whose languages run in parallel columns a language IS a column. Verified in the binary's own output rather than in the source: the line now reads "read where text sits on the page, which is how parallel language columns are found". Still optional, and correctly so. A document probed without it gets its per-page language map and loses only the column resolution, which the probe reports rather than failing on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The comment claimed the probe "roughly doubles and stays under four seconds" from adding two component timings together. Measured end to end instead, best of three with a warm page cache: 560-page, 15 MB manual Analyze 3.71 s, of which run extraction 1.86 s (50%) 68-page, 9 MB manual Analyze 4.09 s, of which run extraction 3.03 s (74%) The doubling holds for the first document. The second contradicts the summary: it is the smaller document and costs more, so "under four seconds" was wrong, and the pass dominates rather than matching its sibling. The one attributable difference is content density — 139 KB and 110 runs per page against 27 KB and 61 — and the comment now says that without claiming it is the mechanism, because the mechanism was not measured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
internal/doc has computed regions since "Store a language that is part of a
page", and nothing stored them: SaveProbe wrote doc_pages and doc_langs only,
and doc_pages has one row per page so a page cannot hold two languages. The
parallel-columns manual's five languages were recomputed on every probe and
forgotten. This is the table that keeps them, plus the write and the read-back.
Additive: a new doc_regions table, no rebuild. 00002 and 00003 are committed and
editing either would diverge from any database created from it. doc_pages stays
as it is, because a page really does have one dominant script, one printed folio
and one tag position; only language is not per page.
KEYED ON GEOMETRY, (document_id, source, page, x0), and the cost of the
alternative was measured rather than argued. Keying on the label instead --
doc_langs' shape, (document_id, source, page, code) -- loses 2 of the real
column manual's 132 regions and 1,977 of its 251,404 characters, because two
German columns on one page are then the same row. That is the concrete breakage
docs/design/regions.md exists to fix, and the test asserts it directly.
x0/x1 are INTEGER although doc.Region carries float64. A float in a primary key
needs two probes to produce bit-identical floats before the upsert converges;
one unit is one pixel of a pdftoppm -r 108 raster, so sub-unit precision
describes nothing about a column boundary. Rounding cannot merge two columns:
over all 169 columns of testdata/fixtures/thomas-drybox-amfibia.json the two
closest x0 values on any one page are 143 units apart and the narrowest column
is 122 wide, both three orders of magnitude clear of the half unit rounding can
move an edge. Rounded, not truncated -- truncation biased three of the
round-trip test's four regions left by one unit.
THE DELETE BEFORE THE INSERT IS LOAD-BEARING. Unlike doc_langs, internal/doc
produces one resolved region set in which source records which signal named each
region, and that attribution changes between probes: a column named by its
alphabet on one run can be named by its printed tag on the next. With source in
the key that is a different row, so an upsert alone leaves the superseded one at
the same x0 and the page reports itself twice. Said plainly in the migration so
it is not tidied away.
Regions are written only when the probe actually read them. A non-empty
RegionNote means positioned text could not be read at all -- pdftohtml absent or
failing -- so existing rows are left alone; poppler is optional at runtime here,
and deleting a good region map because a tool went missing from the host would
be destructive. An empty note means the probe did read the document, so its
answer replaces even when it is no regions, which is the honest outcome for an
encrypted document or one with no text layer.
A SQLC BUG FOUND ON THE WAY, WORTH KNOWING BEFORE WRITING ANOTHER QUERY. sqlc
v1.31.1 -- pinned in tools/go.mod, built to ./bin/sqlc, so this is not whatever
a developer happens to have installed -- tracks each statement's end offset in
BYTES but slices the text in CHARACTERS. Every non-ASCII byte earlier in a
queries/*.sql file silently truncates the tail of every statement after it.
Measured: with one em-dash in a comment, "ORDER BY first_page, code" generated
as "ORDER BY first_page, co"; with four, as "ORDER BY first_pa". The first
version of SummarizeDocRegions here generated as "ORDER BY first_page," -- valid
Go, invalid SQL, failing at PREPARE time inside a background job against a
user's database. make sqlc exits 0 and the linter passes throughout.
All ten pre-existing query files are pure ASCII, which is the only reason this
had never fired; that was checked, not assumed. docregions.sql is therefore
ASCII with the reason at the top, TestQueryFilesAreASCII guards the cause, and
TestDocRegionQueriesExecute executes every generated doc_regions statement
against a real migrated database so a mangled one fails the build instead of a
user's probe. Columns are listed explicitly rather than SELECT *, which is the
better query anyway.
Verified, and every behaviour reverted to confirm its test catches it:
make check green -- go test -race -shuffle=on, golangci-lint 0 issues, tsc.
MANUALBOX_TEST_FIXTURES=1 go test ./internal/registry/ ./internal/doc/ green.
Acceptance, on the real 68-page manual through doc.Analyze -> SaveProbe ->
read-back: 132 regions computed, 132 stored, 251,404 characters preserved
exactly, all five of de/kk/pl/ru/uk surviving, the eight human-verified
pages matching column for column, and a re-probe converging value for value.
Generated SQL diffed against its source statement by statement, because for
this class of bug an exit code proves nothing.
delete-before-insert removed -> stale-row test fails, 2 regions on page 7
RegionNote check removed -> intact test fails, good rows deleted
math.Round -> math.Trunc -> round-trip test fails on 3 of 4 regions
key on code instead of x0 -> real manual loses 2 regions, 1,977 chars
ON DELETE CASCADE removed -> cascade test fails
one em-dash in a query comment-> both generator guards fail
Deliberately not done: nothing surfaces regions through the API or the pre-flight
gate yet, so internal/api and internal/ingest are untouched and the gate still
prices from per-page runs -- that is a later commit. internal/doc is unchanged,
so the sequential 34-section manual reports exactly what it reported before,
which is what regions.md requires of a change that improves the other document.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard and its measurement are right and stay as they are. The stated cause was asserted, not read out of sqlc's source, and it points the wrong way: byte offsets applied to characters would overshoot a statement's end, not cut it short, and what happens is truncation. So the files now state the rule that was measured — one character of SQL lost per extra byte a non-ASCII character occupies, in every statement after it — and stop explaining the internals. Also recorded: the corruption has two shapes, and the quiet one is the dangerous one. Reproduced independently in this checkout by putting two em-dashes in a comment in docpages.sql, where sqlc garbled a statement into tokens like "SELdocument_id" and "_no" and exited noisily. The same character elsewhere generates clean Go with broken SQL. Neither a clean run nor a loud failure tells you a file is safe, so the ASCII rule is the guard, not vigilance. Verified after the edits that the generated code differs only in copied comments and that SummarizeDocRegions still ends "ORDER BY first_page, code", which is the statement that was mangled. CLAUDE.md records the state regions are actually in: computed and stored, verified on both manuals, and NOT surfaced — the gate still prices from per-page rows and CostEstimate.Chars is still documented as always present while never being set. It also names the two open gaps that cost something measurable, so neither reads as an oversight, and warns about the ASCII rule before someone writes an em-dash in a query comment. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The printed-index signal was pure fiction on the parallel-columns manual, and it reached the user. The gate said, verbatim: "68 pages in 2 languages, none of them yours. It has fax and Ukrainian." Page 68 is that manual's back page of service addresses for six countries, and it was the ONLY page of the document the index parser read at all -- its real contents pages, 2 and 3, are laid out one column per language and the parser cannot read them. Every entry it produced was a fragment of an address: VIA from "Via Monte Rosa" title "Monte Rosa" claimed pages 28-45 FAX from a fax label claimed pages 46-48 UA from a Ukrainian address title "Telefax" claimed pages 49-68 Z from "Sp. z o.o." title "o.o. Telefon" NDE from "Neunkirchen" a phone number for its page, 4931 Reconciliation then trusted those claims, so a five-language manual reported two languages, neither of them right, over half the document. The fix needs no new threshold: an index entry's token must NAME A LANGUAGE, not merely be shaped like one. That is the rule KnownLanguage already states for regions, applied one layer earlier -- and the decision it now gates is whether the page is a contents table at all. VIA, Z, NDE and GA name nothing; FAX parses as the language "fax" and is not in the set of languages that appear in appliance manuals. Only UA survives, one entry against a floor of three, so the page stops being a contents table and the whole fabrication disappears. Verified on both documents. The column manual now yields no contents page and no index runs at all, which is the honest answer for a document whose index cannot be read. The sectioned manual is untouched: 34 entries, 34 with titles, its language map, page counts and spans unchanged. This exposed something that had been hidden by the bug rather than caused by it. Page 68 was being suppressed as a "contents page" -- a wrong classification with a convenient outcome -- and without it the repertoire signal names that page Turkish from one of its three columns, the other two establishing nothing. So the refusal is now explicit and for the right reason: a page named by a MINORITY of its columns is named on weak evidence. Measured before adopting rather than after: that describes exactly one page across both manuals, page 68, the one a human checked and recorded as establishing nothing, and no page at all of the sectioned manual. The two refusals are separate guards and both are needed. The contents guard is what stops the sectioned manual's pages 2-5 being named from their own letters; the minority guard is what stops an address page being named from a third of its columns. Neither document exercises both. Both were reverted to confirm the tests catch them: dropping the KnownLanguage check reproduces all five junk entries, and dropping the minority guard names the address page from one column. The fixture's note on page 57 is corrected. It blamed the cell's length for German being read as Finnish -- "a short cell sharing a-umlaut and o-umlaut gives too little to work with" -- which is a distant second cause. The page PRINTS its language: a D in the oval at its top left, inside that column's box, which columnTag finds and then rejects because a single letter needs corroboration from the document's own index vocabulary, and this manual has none. Supply the real vocabulary and the column reads code=D lang=de src=page-tag; the alphabet is never consulted. So the misread and the 53-of-169 tag attribution are one bug, not two, and the manifest now says which one. What this does NOT fix, and it is now the largest known gap: the columnar contents pages still cannot be parsed, so no vocabulary is recovered from that manual and printed-tag naming stays at 53 of 169 columns instead of 79. Recorded in the fixture's known_limitations. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
language-detection.md described the printed-index signal as unreadable on the column manual. That understated it in one direction and overstated the fix in the other, and both matter to whoever reads it next. Understated: the parser did not merely fail on that manual's contents pages. It read its BACK page instead — service addresses for six countries — and took it for the contents table. The table of what it scraped is now in the doc, because the shape of the wrong answer is the useful part: a street prefix, a fax label and a postal country code became languages claiming more than half the document, and a phone number became a page reference. Now accurate about the repair: the junk is gone, because an entry's token must name a language rather than merely be shaped like one. The columnar contents pages are still unreadable, so no vocabulary is recovered and printed-tag naming stays at 53 of 169 columns rather than 79. And it now records the connection that was missed twice: page 57's German cell read as Finnish because the D printed in that page's own corner is rejected for want of the vocabulary. The misread and the attribution gap are one bug. Fixing the parser closes both, which makes it worth more than its size suggests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pre-flight gate is the screen a user decides on, and for the parallel-columns manual it said, verbatim: "68 pages, but no language could be identified." while doc_regions for that same document held, measured from the stored rows: ru 49,526 chars over 26 pages de 47,641 over 26 pl 48,600 over 27 uk 48,317 over 26 kk 46,249 over 26 fi 289 over 1 The gate read registry.LanguageRuns(.., SourceReconciled) -- the per-page doc_langs rows -- and that table has 0 reconciled rows for this manual, because a page there holds three languages and no per-page answer about it can be right. Storing the regions and showing them were split into separate slices; this is the second. So the gate now reads registry.Regions where a document has them and falls back to the per-page runs where it does not. Not either/or, though: where a reconciled run exists it still owns the entry, because a run carries a section title, a source and a confidence that no region stores. The regions add size on top. That is what keeps the sequential manual byte-for-byte identical -- its inScope and its 33 other languages compare equal field for field against the previous binary's output, apart from the new fields being filled in -- while the columns manual, which has no runs at all, gets its whole map from regions. An empty region set is still not the claim "this manual has one language": a document probed on a host without pdftohtml has none stored and a complete per-page map, and Regions' own doc comment says so. CHARACTERS LEAD AND PAGES ARE CONTEXT, which is the decision regions.md records. German is 26 of 68 pages of that manual, 38% by pages, and 20% by characters -- because it occupies one column of each of those pages. Leading with 26 pages would say a household reading German faces 38% of the document when it faces a fifth of it. So every language now reports its characters, its share of the document's named text, its pages as a locator, and whether it SHARES those pages rather than owning them. That last field is what stops the page count misleading, and it is worked out from a page carrying more than one region: testing x0 against zero is not equivalent, because a leftmost column can legitimately begin at the page's left edge, and a test whose left column starts at 0 is what pins the difference. Two fields were declared and never assigned anywhere, both always 0: CostEstimate.Chars documented as "measured, free, and always present" Gate.UnlabelledPages documented as content pages no signal could name Both are now derived from what is stored -- the gate is built entirely from stored results so it survives a restart and costs nothing to render, and nothing here re-probes. Cost.Chars is the in-scope characters: 47,641 of the columns manual, 37,595 of the sequential one. UnlabelledPages must be counted from the regions where there are regions, and this is measurable rather than arguable: counted from doc_pages the columns manual reports all 68 of its pages as unnamed, because it has no per-page language at all, which would send the next reader looking for a statistical detector this document does not need. From its regions it is 2 -- pages 67 and 68, the service addresses at the back that genuinely name nothing. The sequential manual is 0, its printed tab having named all 553 content pages. ScopePages is now DISTINCT pages carrying an in-scope language rather than a sum over languages. A household reading two of the five column languages was told it had 52 pages of a 68-page document, and three would have made 78. Where languages do not share pages the two agree, which is why 16 stays 16. Conflicts deliberately stays counted over runs. The UI explains that number as the document's own contents table disagreeing with its pages; the sequential manual has 1 of those and 32 conflicting regions, which are a different disagreement -- a column's alphabet against the page's printed tab. Printing 32 under that sentence would be a lie. A region's dispute reaches the user as conflict on the language. Both mirrors of the contract are updated. web/src/api/types.ts keeps every field it had, gains GateLanguage extends LanguageRun, and widens LanguageSource, which was missing repertoire and "" -- the columns manual's languages are named by exactly those two, so the previous union was already wrong about live data. docs/api/ openapi.yaml had no Gate schema to extend: it documents only the M0 surface, so the gate path and its four schemas are added rather than edited. No React component changed. The screen still renders pages only, and CLAUDE.md now says so where it previously said the gate was not reading regions at all. Verified where a user sees it, against both real fixtures through the HTTP API on a scratch data dir, with the household set to de. The columns manual: chars 240,622 scopeChars 47,641 scopeCharFraction 0.198 scopePages 26 unlabelledPages 2 cost.chars 47,641 "This manual contains 6 languages across 68 pages. Yours is 1 of them -- 47,641 characters, 20% of the text." Six, not five, and it is not a regression: fi is one 289-character region on page 57, 0.1% of the document, stored by the repertoire signal, which regions.md measured at 93% accurate and explicitly refused to gate with a threshold. Characters-led reporting is what makes it harmless -- it now reads as 0.1% rather than as a one-page language beside five real ones. Filtering it would need the threshold that document says no evidence supports. Five load-bearing changes were reverted one at a time to confirm the tests catch them: reading the runs again reports 0 languages for the columns manual; dropping Cost.Chars fails three tests; counting unlabelled pages from doc_pages gives 4 where 1 is right; summing scope pages gives 8 for two languages sharing four pages; and the x0 rule for sharing calls a flush-left column its own page. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gate is the screen a user decides on, and it was the last place still
counting in pages after the API stopped. For the parallel-columns manual it
rendered, verbatim:
In your languages 52 (76%)
DE German 26 pp · 2–62
Import 52 pages
Every one of those numbers is true and the sentence they form is false. German
occupies ONE COLUMN of each of those 26 pages, shared with four other
languages: 47,641 characters, 20% of the document's text. "76%" is the share of
pages a household's languages appear anywhere on, and reading it as how much of
the manual you get is off by nearly a factor of two — the honest figure is
scopeCharFraction, 40%. A button offering to import "52 pages" when 40% of the
text is wanted repeats the same error at the moment of spending.
So characters lead and pages are context, which is the decision regions.md
records and the API already carries:
German DE 47,641 chars · 20% of the text
appears on 26 pages, sharing each with other languages
sharesPages picks that second line, and it must, because the two layouts are
not the same fact. A sequential manual's section owns its pages and its span is
the whole truth about it; a column shares every page it appears on and a bare
page count about it misleads. Written out, the same component says of the
560-page manual:
German DE 37,595 chars · 4% of the text
"Benutzerhandbuch" · pages 23–38, all its own
The title is kept and given the space it needs. It is the section name the
manual prints in its own contents table, only the printed index can supply it,
and it was previously a truncated fragment squeezed between the name and the
page count.
The two fields the previous commit filled in are now shown. cost.chars is what
the disabled Import button counts, because that struct is the one a caller asks
about spending. unlabelledPages gets a sentence rather than a stat, since "2"
under a label means nothing: pages carrying text no signal could name are the
reason the languages above do not add up to the document, and that needs saying
in words. It is omitted at 0, which is the sequential manual, where the printed
tab named all 553 content pages and there is nothing to explain.
The 289 characters of Finnish are the awkward case and they are NOT filtered.
Regions.md refused a confidence threshold for want of evidence, and dropping a
language from the screen is that threshold wearing a different hat. Two things
keep it from reading as a peer of five real languages instead. Percentages keep
a decimal below 1%, so it renders "0.1% of the text" rather than rounding to
"0%" like a bug or "1%" like an overstatement; and both lists are ordered
biggest-first, so a 289-character misreading cannot sit between two 48,000
character languages looking equal. Ordering by size is also what makes "Also
present:" worth naming languages in at all — the 34-language manual gets its
six largest and a count, where the old collapsed control only said "the other
32 languages in this document".
Verified against both real manuals end to end, not by typecheck: built the
binary, served it on a scratch data dir with the household set to de,uk,
uploaded both fixtures over the HTTP API and read the gate JSON, then rendered
the real GatePanel against that exact JSON with react-dom/server, which is
sound here because the component takes its data from props and runs no effects.
There is no browser automation on this machine, so this is the closest to a
screen it can be checked from; a human still needs to look at it. The columns
manual:
chars 240,622 scopeChars 95,958 scopeCharFraction 0.399 scopePages 52
unlabelledPages 2 cost.chars 95,958
Ukrainian 48,317 · 20%, German 47,641 · 20%, both "appears on 26 pages,
sharing each with other languages"
Also present: Russian, Polish, Kazakh, Finnish
Finnish 289 chars · 0.1% of the text, "appears on page 57, shared with other
languages"
Import 95,958 characters
and the sequential one, where sharesPages is false throughout:
chars 1,047,950 scopeChars 71,866 scopeCharFraction 0.069 scopePages 32
unlabelledPages 0 and no sentence conflicts 1 and its existing sentence
German 37,595 · 4%, "Benutzerhandbuch" · pages 23–38, all its own
Ukrainian 34,271 · 3%, "Посібник користувача" · pages 297–312, all its own
Also present: Greek, French, Italian, Polish, Russian, Hungarian, and 26 more
Import 71,866 characters
The distinction was reverted to confirm the render goes wrong without it: with
placement ignoring sharesPages, the sequential manual claims its German section
"appears on 16 pages, sharing each with other languages" and loses the span
23–38 that is the true thing about it.
Two judgements worth disagreeing with. Shares are integers at and above 1%, so
the 34-language manual reads 3% and 4% where the underlying values are 3.27%
and 3.59%; the exact character count is on every row, and integers agree with
the server's own summary sentence directly above, which is %.0f. And Median
chars/page stays in the stat row: it is the signal for whether a scan needs OCR,
which is a different question from how much a household reads.
No new dependency, no change to the API types, and the stat row keeps the page
figures as a quiet second line rather than dropping them — 52 of 68 pages is
still where the language sits, it is just not the size of it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Converting a manual into headings, paragraphs and tables needs to tell a heading
from a paragraph, and ExtractRuns was throwing away the only thing that can:
parsePDFXML ignored <fontspec> entirely. Each run now carries its size, its family
verbatim, and a weight and slope -- and the interesting part is that the weight had
to come from two independent signals rather than one.
extern.go already recorded why size will not do it. This is the whole distribution,
in characters, measured over both fixtures and printed by the new
TestFontDistributionOfBothManuals:
sequential manual parallel-columns manual
the body size 11: 64.1% of chars 14: 84.0% of chars
the next size up 17: 14.7% 17: 5.5%
...which is set in MiSans, regular FuturaCon-Lig, the body face
heavier at the body size MiSans-Medium 5.1% FuturaCon-Med 17.2%
Row three is the recorded trap, now confirmed whole-document: the larger face is
what the safety text is set in, so "larger than body means heading" promotes 14.7%
of a 560-page manual. Row four is the other half, and it is worse on the columns
manual -- 84.0% of its characters are ONE SIZE, so size discriminates almost
nothing there, and what separates its emphasis from its body is only the weight of
the face at the same size.
WEIGHT IS NOT A NUMERIC ATTRIBUTE IN THIS XML, and the two ways poppler expresses
it disagree in both directions, so both are carried and neither is folded into the
other. Its <b> markup is effectively boolean and draws the line above Medium:
counted per family it wraps every run of a name saying Bold, Demibold, SemiBold or
Xbold -- FuturaCon-Bol 157 of 159, Function-Xbold 81/81, MiSans-Demibold 2861/2861,
Arimo-SemiBold 89/89 -- and not one run of a name saying Medium: FuturaCon-Med 0 of
1,241, MiSans-Medium 0 of 2,887, Arimo-Medium 0/248. It also marks bold where no
name admits it: FuturaBQ 78 of 78, FuturaStd 11/11, Calibri 12/12, and 498 of the
16,426 runs whose family reads plainly "MiSans" -- same base name, different
embedded subsets, different real weight. Slope disagrees too: <i> appears only on
oblique-named families, yet FuturaCon-BooObl gets 66 runs and not one <i>.
What settles it is that the two manuals rely on OPPOSITE signals. On the columns
manual the names carry the document: 93.4% of its characters are in a face whose
name states a weight, while poppler marks 1.5% of them bold. On the sequential
manual it is the reverse -- 73.2% of its characters are in a face whose name states
nothing at all (MiSans, MiSansLatin, Sarabun, Arimo, HarmonyOS_Sans_Naskh_Arabic),
and the markup is the only weight there is. Either signal alone reads one document
and is close to blind on the other, and there are two documents. So the name gives
a graded Weight rather than a bold flag, because collapsing Medium into "not bold"
discards the single most useful distinction in the columns manual, and poppler's own
verdict is reported beside it as MarkedBold/MarkedItalic. No rule here decides what
a heading is; that is the next piece and it needed this data first.
THE FONTSPEC TABLE IS DOCUMENT-WIDE, WHICH IS THE OPPOSITE OF WHAT THE PER-PAGE
<fontspec> ELEMENTS SUGGEST, and it was measured rather than assumed. Poppler
26.07.0 allocates ids once per document and declares each on the page that first
uses it: 83 ids, 0-82 contiguous, over 29 of the columns manual's 68 pages, and 309
ids, 0-308, over 167 of the sequential manual's 560 -- with not one id redeclared
anywhere in either document. Every later page refers back. A table scoped to a
single page therefore resolves nothing for most of a manual: it leaves 6,201 of
7,493 runs and 32,858 of 34,413 runs, 83% and 95%, with no font at all while their
coordinates and text arrive intact, so nothing else fails and nothing says so.
Reverted to per-page, the new unit test reports size 0 on page 2 and the fixture
test names both of those counts. A page that does redeclare an id still wins for its
own runs, since its declaration overwrites the entry before its text is read;
checked on both documents, poppler emits a page's fontspecs before any of its text.
Two more things nothing would have guessed, both now asserted. The fontspec size is
in the same 1.5-scaled space as the coordinates, not the PDF's points -- 11pt and
17pt text written into a generated PDF come back as 17 and 26 -- so it is directly
comparable to a run's Height, and a caller must not print it as a point size. And
an absent font attribute cannot read as id 0, because poppler numbers ids from 0
and that is a real font; removing the sentinel makes a run with no attribute inherit
a stranger's semibold, which is exactly the silent kind.
The custom UnmarshalXML keeps its measured behaviour untouched -- 355 wrapped runs
of the columns manual whose text `,chardata` returns nothing for, the printed D, PL
and UA tabs among them -- and the same walk now notes which element wrapped it. Only
a wrapper enclosing the WHOLE run counts: all 355 of those and all 4,336 of the
sequential manual's are wrapped whole, none partially, and labelling
`plain <i>word</i> plain` italic would name a line after one word of it. Nesting is
real, 5 runs are <i><b>...</b></i>, so both are recorded at any depth.
testpdf grew a second face because everything it wrote was Helvetica at one size, so
no generated document could vary either and the poppler-driven test would have had
to become fixture-only. One more standard-14 font object, no embedded file. Its
limit is measured and stated: poppler names both faces "Helvetica", since a standard
font is not embedded and has no subset name to read a weight out of, so that test
covers size and the markup, and the family-name weights are unit-tested against the
real families of both manuals instead.
TextRun gains a field and loses nothing: X, Y, Width, Height and Text keep their
names and their order, Font comes last, and its zero value means "not known" so the
hand-built literals in columns_test.go, columnlang_test.go and regions_test.go are
untouched and still mean what they meant. The columns manual still finds its 169
columns and still names them 53 by printed tab and 114 by alphabet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pattern regions.md set: the decisions go down first, from measurement rather than from memory, so the reasoning is not buried in a chat log. Conversion is the first stage that produces something a user reads rather than decides on, and it is the first that is not free. Blocks, not a page image and not a viewer. That is forced by what comes after rather than chosen: ingest.md requires extraction to cite a paragraph rather than a document, and search wants the same units. A page image satisfies neither. Four decisions rest on numbers that were measured for this, two of which contradicted the obvious approach: A heading is found by weight and by LENGTH, not by size. On the sequential manual 17pt regular is 11.4% of the document at 70 characters per run - safety body copy, which "larger than body means heading" promotes to a heading. Its real headings are 15pt semibold at 15.5 characters per run. Characters per run is the discriminator. Both weight signals are needed because the two manuals disagree about which one exists. The column manual states a weight in the face name for 93.4% of its characters; the sequential manual's faces state nothing for 73.2% of its own, and poppler's <b> marking is the only weight it has. Tables come from the ruled lines, which no tool the probe uses reports at all. pdftocairo gives them exactly. Recovered against renders: 12 of 12 cells, 16 of 16, 32 of 32, 25 of 29, 37 of 47 - with both sets of misses characterised rather than averaged, the first being header rows whose top border is not drawn and the second being vertically merged cells. The text guard is not optional. "Has ruled lines" fires on 68 of 68 pages of the column manual. A table shape leaves 13, and three of those are grids of framed illustrations - geometrically identical to a table. 14 of their 15 cells hold no characters where every cell of a real table holds text. With both guards: 10 pages and 170, the latter being 34 languages times 5 table pages exactly. Cost is smaller than it first looks and is stated so nobody re-derives it: 8.6 s and 42.3 s over WHOLE documents, but conversion runs only over the pages in scope - 26 of 68, 32 of 560 - and after the gate, so it never slows the free probe. Six things are deliberately not solved and say so, the largest being that a table with no ruled lines is invisible. The column manual's pages 62-66 are exactly that, and a text-only signal was looked for and not found: row alignment points the WRONG way, since three parallel translated columns align at 67-100% while a table's cells align at 29-40% - a translated paragraph corresponds to its neighbour and a two-line question does not correspond to a ten-line answer. Two corrections to what the column fixture recorded, both found by measuring it: its tables are on pages 52-61 and not 57-61, and its pages 62-66 are unruled tables that nothing had recorded at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
conversion.md set the contract; this is the piece under it. One region of one page
becomes ordered blocks - heading, paragraph, list item - and nothing is stored yet.
Table and figure are declared as kinds and never produced, so the ruled-line work
has a vocabulary to emit into without changing a database column later.
The block is keyed on the page, the region's left edge and the index within the
region. Same reasoning as doc_regions and the same reason: a job handler can run
twice, and a surrogate ID would make the second conversion insert a parallel set
instead of converging. TestBlocksConvergeOnASecondRun holds it on a real document.
THE CONTRACT IS WRONG ABOUT WHERE READING ORDER COMES FROM, and this is the largest
thing settled here. It says order comes from the region, and for a parallel-columns
page it does - a boxed region IS one text column. But regions.md rule 3 deliberately
stores a page of several SAME-language columns as one whole-page region, so on those
pages sorting inside the region is exactly the pdftotext -layout interleaving the
contract names, committed under another name. Measured on the column manual's page
62: a whole-page German region holding two prose columns whose baselines do not even
line up across the gutter (102 against 102, then 118 against 120). Reverting the fix
welds them into "rial bitte umweltgerecht. sich bei gewerblicher Benutzung oder
gleichzusetzender Beanspruchung..." and "* Verpackung Unabhaengig von den
Gewaehrleistungsverpflichtungen...". The sequential manual has the same shape on the
199 pages that read as three columns. So a region is subdivided by DetectColumns
first, and order is column by column, spanning runs read first.
The heading rule needed three corrections, all from measurement, two of which
removed a test rather than adding one:
A SIZE FLOOR IS WRONG, which is the opposite of what it looked like. The sequential
manual's safety pages are set entirely in 17pt, so 17pt is their body, and their
real subheadings - "Nutzungsbeschraenkungen" on page 23 - are 15pt Demibold:
SMALLER than the text they head. Requiring a heading to be at least the body size
loses 80 of them and guards nothing. The small bold text it looked like it was
guarding against is 9pt Demibold at 8.0 characters a run on 215 pages, and every one
is "Note:" or "Hinweis:" opening a paragraph - a lead-in run, not a line, already
excluded because a line takes its face from its dominant characters.
A HEADING MUST START A BLOCK. The last line of a paragraph is short by definition,
so any paragraph in a face heavier than the body hands its final line over. Not a
corner case: the column manual sets 17.2% of its characters in FuturaCon-Med at the
body size, and per-line classification produced 280 headings reading "Umgebungen
benutzt werden.", "gung durchgefuehrt werden.", "vo vzryvoopasnykh pomeshcheniyakh."
Requiring the space above a heading that the typesetter put there removes all 280 and
costs nothing measurable.
A HEADING CONTAINS A LETTER. The column manual's page 11 is an exploded diagram whose
26 numeric callouts are FuturaCon-Med at 17pt - larger AND heavier than that page's
body, two characters long, so they pass every typographic test. A callout, a folio
and a chapter number are numbers; a heading is words.
THE LENGTH SIGNAL HAS NO THRESHOLD TO FIND, and saying so is more useful than a
number that looks measured. Histogramming every candidate's share of its measure
gives a smooth continuum from 5% to 100% on both manuals - the column manual's 632
candidates run 33 at 60-64%, 25 at 65-69%, 116 at 95-99% - with no trough anywhere.
Counting runes, which is what the contract states, is no better: 135 at 50-59 runes
against 133 at 20-29, again no valley. The reason is real: a manual sets one-line
paragraphs and two-line headings, so the populations genuinely overlap. 0.6 of the
measure is therefore a soft cut chosen for precision, because a false heading is
visible wrong furniture while a missed one degrades to a paragraph and still reads.
Its cost is stated rather than hidden: a heading filling a narrow column comes back
as a paragraph, which is what loses "Fehlersuche", "Feilsoeking" and "Depanare".
Everything else derives its reference from the region rather than assuming one. The
body face is the region's own most-common size and weight by characters, which is
what makes a page of nothing but 17pt safety copy judge against 17pt. The line pitch
is the MODE of a column's own line gaps, not the median, and that was forced: page
62's left column has nearly as many break gaps as body gaps, so its median is 18
against a real pitch of 16, high enough that the 20- and 21-unit breaks it must find
fall inside the factor and are lost. The mode is 16. Both weight signals are read,
for the reason the Font commit measured.
Lines come from sameBaseline, extracted out of sharedBaselines in columns.go so the
rule has one definition rather than two that drift. No signature changed there.
Acceptance, checked against 108 dpi renders of the column manual's pages 62 and 14
and the sequential manual's pages 23 and 24 rather than against counts:
page 62 4 headings, 4 bulleted disposal items, 6 numbered guarantee clauses,
9 specification rows each its own block - block for block with the render
page 14 German is the MIDDLE column of three; 3 headings, 3 bold step captions,
4 warning paragraphs, and no Polish
page 23 1 heading, 1 subheading, 1 paragraph, 7 bullets
page 24 1 heading, 12 bullets
Document-wide: the column manual's German gives 435 blocks over 26 pages, 47,355
characters, 54 headings, with zero Cyrillic letters and zero Polish-only letters in
any of them - the negative the contract calls the one failure a reader would notice.
The sequential manual's German gives 466 blocks over pages 23-38 with no Greek,
Arabic, Hebrew or CJK in any of them.
Five rules were reverted one at a time and each has a test that fails:
size-only headings promote the display line and lose the real one; a heading not
required to start a block promotes a paragraph tail; sorting inside a region
interleaves two columns; no hanging-indent rule orphans every guarantee clause's
body; and no tab-gap test folds page 11's nine parts into one paragraph.
Deliberately not solved, and recorded in the file: page furniture is not identified,
so the printed "DE" badge comes back as a heading on 110 pages and the folio as a
one-character paragraph - what identifies furniture is that it repeats in the same
place across pages, which is a comparison this pass cannot make. Hyphenation is not
undone, because German legitimately ends a line with a hyphen ("Ein- und
Ausschalten") and telling that from a broken word needs a dictionary. A two-line
heading set with generous leading becomes two headings. A table still is not a table.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three errors in conversion.md, all found by building the thing rather than by reading it again, and the first is a real design mistake rather than a slip. READING ORDER DOES NOT COME FROM THE REGION'S BOX. The contract said to sort runs down-then-across within a region, which is the mistake it spends a paragraph warning against — because a region is not always one column. regions.md rule 3 deliberately stores a page whose columns are all the SAME language as one whole-page region, and the column manual's page 62 is two columns of German at x=43-443 and x=463-863 stored as a single region spanning 0-892. Sorting inside that box interleaves them line by line, producing "rial bitte umweltgerecht. sich bei gewerblicher Benutzung oder gleichzusetzender Beanspruchung…". Verified against the stored regions: page 62 is one row, 0-892, de. The sequential manual has the same shape on the 199 pages that read as three columns. A region must be subdivided by DetectColumns first. Rule 3 is still right — a page of same-language columns is one language territory — which is what makes this a seam rather than a contradiction. The region says which language and how much text; the columns inside it say in what order to read. There is no size FLOOR for a heading either, which is the opposite of how the contract read. The sequential manual's safety pages are set entirely in 17pt, so 17pt is their body there, and their real subheadings are 15pt semibold — smaller than the text they head. A "not smaller than the body" test loses 80 real headings and guards nothing. And its 17pt safety copy is not "regular": that face is plainly MiSans and states no weight at all, so it reads as unknown, which sorts BELOW light. Any rule comparing weights has to know that. The unruled specification table is not five pages. It is one table repeated per language, a block within the disposal-and-warranty page: 62 German, 63 Polish, 65 Ukrainian. Also records what the first half settled and could not: page furniture cannot be identified from one page (the printed DE badge reads as a heading on 110 of them, and the sequential manual genuinely titles sections A, B and C — what identifies furniture is repetition across pages, a different input); no gap factor separates every paragraph, since 17-unit gaps occur both within and between paragraphs on one page; the pitch must be the mode rather than the median, because that page has nearly as many breaks as body lines and its median is 18 against a true pitch of 16; and a heading's share of the measure is a smooth continuum with no gap to put a threshold in, so 0.6 is chosen for precision and loses headings that fill a narrow column. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…moved Two fixture pins were left asserting numbers the code stopped producing: the columns manual's German conversion reads 460 content blocks where the pin said 431, and the sequential manual's German 449 where it said 448. They pass in isolation only if you run them before the change that moved them; they were red in the full suite and I nearly filed that as parallel-test contention. It is not contention and it is not flakiness — the count is stable at 460 across three runs. The +29 is text that was always content and was welded into the wrong blocks: page 11's parts list arrived as two run-together blocks with the diagram's callouts spliced mid-sentence, and page 57's troubleshooting table now gets its own header rather than one welded from both columns'. Higher is better here, so both failure messages say so. The number alone cannot: this file now carries 432 / 427 / 443 / 431 / 460 for one conversion, and a reader who sees only the value cannot tell a recovery from a regression. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NL8EwtKTPfkzayvokjzbxV
A reader could not reach the sequential manual's exploded parts diagram. It
is on PDF page 5, its four sub-drawings ARE found by the figure pass, 31
places in the content pages say "see A-1", and page 5 falls inside no
language region -- so no conversion has ever served it.
Measured first, on both fixtures, because the brief invited stopping if the
set were junk:
neutral pages chars figures
sequential manual 7 1,656 61
parallel-columns manual 2 11,256 0
The set is worth taking on one document and is furniture on the other, and
the character count points the WRONG WAY between them -- so the picture
count is measured and stored rather than left out.
doc: NeutralPages() from the region map, a bounded lazy ink census
(7 spawns and 2, capped at 32 pages), ConvertOptions.IncludeNeutralPages.
db: 00007 adds doc_pages.figures (nullable -- nil is not 0) and
documents.include_neutral_pages. Additive, so 00006's FTS triggers and
the external-content index are untouched.
ingest: Gate.Neutral from stored rows only; Approve takes one boolean and
the argument for why answering an offer is not composing a scope.
Nothing changes for a household that does not opt in: ConvertOptions' zero
value is today's behaviour and verify.ConvertAll passes it deliberately.
…sion
The set is pinned as CONTENTS and not only as a count, because the brief
asked for the finding either way:
sequential 7 pages -- cover, 3 contents pages, TWO DIAGRAM PLATES, colophon
59 of its 61 figures on pages 5 and 6; page 5 IS A-1
columns 2 pages -- a print code and service addresses in 12 languages
11,256 characters and NOT ONE PICTURE
Opting in: Russian figures 65 -> 126, page 5 = 31 drawings and page 6 = 28,
and all 61 are Neutral, so a German household off the same bytes sees them
too. Opting out: blocks and figures identical, and every block compared
field by field rather than counted.
One stale number corrected rather than copied. CLAUDE.md and conversion.md
say Russian is "431 content blocks"; 431 was the COLUMN manual's German
before two-strip reading, and Russian measures 449 and 58 on this commit.
Generated rather than fetched, so this runs in the default suite: a
front-matter plate of four drawings labelled only "A-1" and "C-5", then
tagged language sections. The plate carries 7 characters, which is
deliberately below MinTextChars -- it exercises the arm of carriesContent
the character floor cannot reach, and a real plate's labels are inside its
drawings.
Five tests, covering the offer, its negative, and the seam:
- the gate offers page 1 with its 4 pictures, read from doc_pages
- a document whose every page is tagged is offered nothing at all
- approving with the flag stores it on the row, the job reads it back,
and FiguresByLang("de") serves the plate
- approving without it converts no block and no figure from page 1
- a yes to an offer that was never made stores nothing
40 strokes per drawing, not 12: the first attempt found 0 figures because
12 is under minFigureInk, which is what made the gate correctly offer
nothing.
The Import button was disabled with "Conversion arrives in the next slice" and api.approveDocument did not exist, so the gate screen has never been able to approve anything -- an opt-in checkbox above a dead button would have verified nothing. Both are wired up. The offer leads with PICTURES where every other row on this screen leads with characters, and that inversion is deliberate and measured: the sequential manual's 7 unowned pages hold 1,656 characters and 61 pictures, the columns manual's 2 hold 11,256 and none. Leading with characters here would invite declining the diagram plates and accepting service addresses. Three states, because absent and zero are different: pictures counted and present leads with them, counted and zero says so plainly and suggests checking the original, uncounted carries the note instead of a 0 that would read as "none". Unticked by default even when it holds the pictures -- defaulting it on would convert pages nobody chose. Ticked and labelled "already imported" for a document approved with them, so a reload reports the real decision. The button's label names the extra pages while they are ticked. pageRanges is its own module so node --test can reach it: 7 tests, and "1-6 and 560" is the sequential manual's real set.
Fifteen mutations, four survived, and each survivor was a real hole: M1 changed one of TWO copies of the content rule and only one of the two tests that should have failed did. The gate and Convert each had their own. Now one exported doc.CarriesContent, which is the same single-sourcing MinTextChars' own comment already argues for -- 'two floors would disagree about the same document'. M2 deleted the ownership test in NeutralPages and nothing failed: converting a 560-page manual would have meant rendering all 560. Now a pure unit test. M7 summed a picture census that had covered only some of the offered pages. An understated count is the error that makes a user decline the pages worth taking, so it is withheld -- now pinned by reaching neutralScope directly, since a partial census needs pdftocairo to fail on exactly one page. M10/M11 were the HTTP seam: nothing tested that the approve body reaches the stored scope, or that a malformed body is refused rather than downgraded. M13 (census bound) and M15 (a language invented for the neutral pages) also survived and are now covered. M15 needed the hermetic fixture changed: the plate has 7 characters and produces no blocks either way, so the generated manual grew a SECOND unowned page of prose. One page could not do both jobs.
conversion.md gets the finding under the passage that recorded this as deliberately unbuilt: a tenth of the page picker is built, it is the tenth containing A-1, and the table shows why the picture count is measured at all -- characters rank the two fixtures' sets in the wrong order. ingest.md gets the gate's second scope and the approve argument. CLAUDE.md gets both, and one correction: it claimed the sequential manual's Russian is '431 content blocks'. 431 was the COLUMN manual's German before two-strip reading; Russian is 449 plus 58 furniture, and the file now says so rather than leaving the next reader to trust it.
make fmt runs prettier over the whole web tree, and several files predate it, so my run reformatted Devices.tsx, Home.tsx and two union types in types.ts that this change never touched. Reverted -- web/ is now additions only apart from the Import button this replaces. There is no prettier gate in CI, so those files were passing as they were and reformatting them here would only hide the real change. Also fixes gofmt on registry/documents.go and a gocritic evalOrder in the neutral fixture helper. golangci-lint: 0 issues.
The idea was to replace carrying a label as text with one merge criterion here: if two drawings' label claims interleave, the page laid them out as one plate, so serve them as one picture. Measured on both manuals and rendered, it does not survive. Page 521, the case it is for, cascades. Merging the pair the diagnosis names does produce one crop holding both drawings and all 20 of their labels uncut -- a real improvement -- but figure 1's want box also clips figure 2's by 14x5 units, so the transitive closure takes all three into a crop 0.629 of the page holding two columns of button-description prose, cut mid-line. Page 522 does not cascade, and that refutes the brief that proposed this: seven printed pictures come back as 9 figures, 6 of which do grow, and only one pair merges. That pair is the worse failure -- the base station's front view in the left column joined to its cutaway in the right, 328 units apart, carrying a fragment of the page's note line and still clipping labels at its right edge. No threshold separates them. Sorted by want-box overlap the target pair sits at 0.148 with plate pairs at 0.173 and above and page 522's at 0.063, and the gap between the drawn boxes orders the two cases the wrong way round: 276 units for the pair that should merge against 328 for the pair that must not. The literal reading of "a shared label field" -- a run claimed by both figures -- has a real signal under it and is worth exactly 13 labels on PDF pages 5 and 6, 0 on any page the reported defect is on. Its floor of two is a 2..3 plateau, which mutation testing found rather than the sweep. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NL8EwtKTPfkzayvokjzbxV
The one-file alternative to carrying a label as text, with the table, the two pages that decide it and the reason no threshold separates them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NL8EwtKTPfkzayvokjzbxV
…to them A crop is a rectangle and a diagram's labels are not arranged in one, so growToLabels could only reach a label by taking in everything between. Page 521 of the sequential manual held 23 of its 34 labels whole; 41 of that document's figures held 88 labels their crops never reached. Figure.Labels carries each claimed label as text with its box as a fraction of the crop, so a reader can place it without knowing the page. Growth is off -- the two together would print every label twice. The claim rule does not move: claimLabels, by terminator plus continuation, is reused unchanged. What figureLabels keeps is growth's CONSERVATIVE half and what it drops is the cutting half, and the columns manual is why the first is not optional: all 9 of its claims are false -- a cover title and eight lines of German body prose -- and what refuses every one is labelExtent finding no clean region, not the crop. So the side is gated and then all of its claims are carried whole, because a text run has no rectangle to cut. Measured over both whole documents, claims carried: 0 and 268. Page 521 goes from 23 held whole to 34, which is every label its crop ever reached.
Page 521 read its label "Датчики перепада высоты" as a floating paragraph, so with the reader also drawing it beside the picture every label would arrive twice. The cheap version -- drop the finished block whose text equals a label -- was measured and does not work. Of the 89 labels on the sequential manual's Russian pages, 23 are a block of their own and 66 are merged into a bigger block: page 522's "Бак для отработанной воды" and fifteen more sit inside larger paragraphs, because a label column sets its lines at the body pitch. Dropping those blocks would delete the neighbouring content. So the run leaves before anything is grouped, which is what splitFurniture records for the language tab. Block.Callout is a SECOND flag rather than reusing Furniture, and the difference is coverage: verify.checkCoverage does not count furniture, because furniture is discarded and counting it would hide a rule that wrongly claimed a paragraph. A label is relocated, not discarded, so its characters must stay in the numerator or coverage could not tell relocation from loss. Russian content blocks 449 -> 409 with 89 callout blocks carrying the text. The growth tests move to grownGuards rather than being deleted: two of them asserted "nothing grew" and would have gone on passing against a pass that cannot grow. TestALabelOutsideTheFinalCropIsTheResidual is replaced by TestALabelReachesAReaderWhole -- 159 labels reach a reader on served pages against 99 the grown crop held whole, and 0 on the columns manual, whose nine claims are all false. The plate-merge refusal keeps its numbers, measured from grownGuards, because it compares against the world it was asked in.
Migration 00008 adds one table. No existing table is rewritten, so 00006's three FTS triggers over an external-content index are untouched and nothing reindexes -- which is the constraint that shaped this rather than an accident. A sixth doc_blocks kind was the obvious alternative and is exactly what 00005's header forbids: widening the CHECK on `kind` costs dropping those triggers and reindexing, with 00003 as the precedent. A label is not a block for a better reason than the cost, though: a block has a place in reading order and a label does not -- it belongs beside a picture. The position is stored as FRACTIONS OF THE CROP, which is the one place the 1.5-scaled house style is broken on purpose. doc_figures' rectangles are compared against each other so they must share a space; a label is compared against nothing, and its only consumer draws it against a rendered image whose size it chose itself. There is deliberately no CHECK bounding the fractions to 0..1: a label sits OUTSIDE the picture, so such a bound would reject every row this table exists to hold. Composite foreign key onto doc_figures(document_id, page, idx) with ON DELETE CASCADE, so a picture a re-conversion no longer finds cannot leave labels behind to be served against whatever figure later takes that index. registry.Figure carries them, so the conversion response serves them with no API change. Read as two queries rather than a join, which would repeat every figure's fourteen columns once per label.
… gets wrong The reader half. FigureView places each label against the crop using the fraction internal/doc carries, so a leader ends in a word instead of in nothing. One DOM, two layouts: the labels are an ordered list, which is what they are, and above `md` each item is lifted to absolute and placed. Below it the list flows under the picture. Nothing is duplicated and nothing is hidden, so a screen reader and a narrow viewport get the same content in the same order. The degradation is deliberate -- twelve labels around a 300px picture cannot be placed without overlapping, and overlapping labels are worse than a list. Placement is PHYSICAL left/right, the one exception to this screen's logical-offset rule, and it is recorded where that rule is stated: a label printed left of a drawing is left of it on a right-to-left page too, because mirroring it would move it off the part it names. The label's own text stays direction-aware. Two things found by testing rather than by review: A one-ULP error in reconstructing a run's position flipped its rounding bucket. 55 came back as 54.999999999999986; 55/2 is exactly 27.5 and rounds to 28, while 27.4999... rounds to 27, so the label silently stayed in the prose. Latent on both fixtures -- real coordinates rarely halve onto a .5 boundary -- and caught by a hermetic test using x=55. Callouts now scan with an explicit tolerance, which has no boundary to straddle. 7 claims are lines of prose, not labels: page 529's numbered step 6 and its continuation, and five more over 531, 546, 550 and 553. NO GUARD IS ADDED, because there is no threshold to put one on -- sorted by length the claims run 37, 38, 39, 40, 42, where 38 is a real label and 39 is a Japanese sentence, and only 2 of 268 end in a full stop. TestASentenceShapedClaimIsTheResidual pins the count and the one-rune non-gap, so a change that opens a real gap is noticed.
The picture inside the positioning box is absolutely positioned, so it is out of flow and contributes nothing to that box's intrinsic size. Where the surrounding layout hands a width down -- a reading column, which sets flex-grow and flex-basis -- that is invisible. A STRIP sizes its children to themselves, so with nothing in flow the box computed to zero width, the picture rendered at 2px and every label of a figure collapsed onto one point. The front-matter plates are laid out as strips. All 109 of their labels piled up illegibly -- "61825", "2620" -- while page 521, which is laid out in columns, looked perfect. Measured in Chrome after the fix: page 5 draws 96 labels with 5 overlapping pairs and none degenerate, page 6 draws 13 with none, and page 521 is byte-for-byte the layout it already had. The width is the stored pixel size times the extended box's share, so a picture with no labels renders exactly as before and one with labels reserves the room its labels need. max-w-full keeps it inside a narrow column. Also gives the narrow-viewport list an interpunct between items: with the paper's arrangement gone, nothing separated one label from the next and a two-line label read as two labels.
A callout is counted by coverage and a furniture block is not, and the two are one line apart in checkCoverage; a callout is exempt from the reading-order check and content is not. Each test asserts the contrast as well as the case, so neither can pass because the geometry stopped being a violation.
Deleting figureLabels' sort left every test passing. The sort is what makes the
stored index stable, so an order that followed the order the page's runs happen
to be listed in would rewrite the same labels onto different rows on every
re-conversion. TestLabelOrderIsStableWhateverOrderTheRunsArriveIn lists the runs
in the opposite order and asserts both the result and its independence.
And withLabels keying on the page alone survived, because every label in the
round-trip test was on figure 0 -- where {page, 0} and {page, index} are the same
lookup. There is now a second labelled figure at index 1, so the key has
something to get wrong.
conversion.md keeps the growth section as history -- it is the number the replacement had to beat -- and adds what replaced it: the claim rule unchanged, the conservative half kept and the cutting half dropped, 0 and 268 claims carried, page 521 from 23 to 34, and the Chrome reading of 34 rendered with 0 clipped. Three defects are recorded rather than left to be found: 7 prose claims with no threshold to refuse them, a wrapped label carried without its later lines, and the brief's own example still floating because its side is correctly refused. Two earlier claims are corrected where they went stale: Rect and InkRect are now equal on every figure, and Russian's block total reads 431 -> 449 -> 409.
zzscratch_reach_test.go was the throwaway that measured how far outside its crop a label sits, looking for a threshold that would separate the plates from the content pages. There is none -- the plates reach 1.18 crop-widths and the content pages 3.91, which is the wrong way round -- and the real cause of the plate pile-up was a zero-width container. The measurement served its purpose and is recorded in conversion.md; the file was swept in by a git add -A.
Blocks 16,132 -> 16,201: the total went UP by 69 while content went DOWN, and both halves are one fact -- 157 runs became Callout blocks and 66 of the 89 on the Russian pages had been arriving inside a bigger paragraph. Nothing left conv.Blocks, which is why coverage stays at a median of 0.996. Clipped figures 24 -> 25, which is the same arithmetic read backwards rather than a regression: the crop is the drawing exactly again, so a leader running past the drawn extent is outside it. The labels those leaders point at are not lost with them. Reading order 24 -> 23, and the finding that left was never a defect: two callout labels of one drawing were being compared against each other.
The constant reads as live and is not: defaultGuards sets growth to zero, and only TestGrowSweep's grownGuards restores it. A reader arriving at the sweep numbers should know they measure the replaced pass.
…ced it The old test counted 14 grown figures and warned in its own comment that if that number were ever zero, the counts above it would no longer say anything about growth. It is zero. So the count becomes the invariant instead -- Rect equals DrawnExtent on every figure of both documents -- which is free, and which a pass that widens a crop again would break loudly rather than silently. The hazard itself moved from the rectangle to the text, and is recorded rather than checked. A label is claimed from a corridor around a drawing, so on a parallel-columns page a figure sitting near a column edge could claim a run from the neighbouring language's column and show it to a household that does not read that language. Neither fixture can produce it: the only document with columns carries no labels, which is asserted here, and the one with labels has whole-page regions. A check would be vacuous, so the note says so and names what would test it -- a third document with columns AND callouts. Renamed to TestTheCropDoesNotChangeWhichLanguageAPictureBelongsTo, since growth is no longer what it is about.
TestTheReportedPageKeepsItsCalloutLabels is named for the fault the user reported, so it is the test that should say what fixed it. It asserted the three grown crops on page 521; it now asserts that the crop IS the drawn box and that the page carries 9 + 11 + 14 = 34 labels, which is the number the change exists for: the grown crop reached 34 and held 23 whole. It also pins the cost of the conservative rule by name. Figure 0's right side is refused because its corridor holds five lines of bullet description, so neither "Кнопка сброса" nor "Датчики перепада высоты" is carried -- the second is the example the brief asked about, and it still reads as a floating paragraph. The same rule refuses all nine of the columns manual's false claims, so this is what that safety costs on a page where it is wrong. `boxed` and `labelsTakenIn` go with the pass they measured: both existed to count "reaches the run" against "holds all of it", and a text run has no rectangle, so there is no partial case left to count.
The user photographed page 521 with `Монтажные отверстия для` drawn on the underside drawing and `держателя`, `насадки для` and `швабры` floating as three separate paragraphs beneath it. A label is a unit and half of one is worse than either answer. The tail was lost to the BAND, not to the gate. claimLabels asks runBeyond, which requires a run to sit within the extent the drawing occupies on the other axis -- right for a leader, which points out of the drawing, and wrong for a wrap, which goes downward: that label is level with the drawing's foot, so its lines 3, 4 and 5 sit 8, 20 and 32 units BELOW the box and were invisible to the continuation fixpoint. Its first two lines were level with the box and were carried. So the loss was between claiming and claiming, not between claiming and carrying. The continuation pass now asks runInCorridor, which is runBeyond without the band, and so does continuesLabel's scan for a run sharing the candidate's line -- both halves of that test must see the same set, and the band-free set is the larger one, so the aloneness test gets stricter rather than weaker. Nothing else moves: a continuation must still be flush to 3 units with a claimed line, adjacent to it to 6, alone on its own baseline, and reachable along a chain that starts at a leader. Measured over both whole documents. The columns manual carries 0 before and 0 after, with all nine of its false claims still refused, which is the constraint. The sequential manual goes 268 -> 276 and 159 -> 167 on pages a reader is served, the plates unmoved at 109. Every one of the 8 is a later line of a label whose earlier lines were already carried -- page 521's three, `Бак для | чистой воды` and `Вентиляционное | отверстие системы автоопорожнения` on 522, `モップパッドホル | ダー取り付け穴` on 542 -- and the eighth is the second line of page 546's bullet caution, which is one of the 7 prose claims already counted and is now wrong whole instead of wrong by halves. TestNoLabelIsCarriedWithoutItsLaterLines asserts the invariant rather than the total: no run left in the flow continues a carried label, except what two bounds cut, and both are measured -- 0 refused by labelCorridor on either document, and 2 by minWrapRunes, the digits "4" and "2" on the plate pages, which are their own numbered callouts and are asserted by name so a real truncation cannot hide in the allowance. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NL8EwtKTPfkzayvokjzbxV
…rnatives The deferral "a wrapped label can be carried without its later lines" is gone from both files because it is fixed, and the guess in it is corrected: it blamed continuesLabel and the cause was runBeyond's band one line further out. The other half of the same report is a judgement, and it is written down with the measurement under it rather than as taste. labelExtent is still load-bearing with nothing being widened: admitting page 521 figure 0's right side draws the three bullet lines under `Кнопка сброса`, which a continuation with no direction chained upward from `Индикатор Wi-Fi`; and a per-chain band gate -- built and run over both whole documents -- carries eight lines of German body prose on the columns manual's page 22, which is the document whose 9-for-9 false claims are the recorded safety property. Captioning them instead fails for the same reason and not a different one: it needs the refused set to BE labels, and 9 of 9 on the columns manual are not. So the leftovers stay in the prose, the cost is stated -- four label-shaped lines adrift on page 521 -- and what would change the answer is named: the leader LINE rather than its end mark, which onPageInk drops because a horizontal hairline has zero height. Probed at 8 of page 521's 26 claims with a first cut, which is why it is written down and not built. Every number that moved is carried as a sequence, because two of them have meant three different things: 23 -> 34 -> 37 on page 521, 268 -> 276 carried, 159 -> 167 served, and Russian content blocks 449 -> 409 -> 404 with callouts 89 -> 95. The 6 runs that left the flow took only 5 blocks with them, which is the 66-of-89 measurement seen from the other side. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NL8EwtKTPfkzayvokjzbxV
268 now appears one paragraph apart from 276 and means the denominator of an older measurement. The full-stop count is 2 either way; only the denominator moved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NL8EwtKTPfkzayvokjzbxV
…on that nothing catches The all-34-languages pin goes 16,201 -> 16,203. Eight runs across four languages joined their own label and only 2 new blocks came of it, because 6 of the 8 were already inside a bigger paragraph -- the 66-of-89 ratio holding at a much smaller sample, which is the shape to expect rather than a surprise. The mutation table is in conversion.md, and its last row is the useful one: putting the band back on the ALONENESS SCAN alone, leaving the candidate found without it, breaks no test on either manual. The change is kept with that written on it in continuesLabel -- both halves of one test must see the same set, the band-free set is the larger one so the test gets stricter, and a run outside the band sharing a later line's baseline is a shape neither fixture prints. It is consistency, not a measured gain, and calling it measured would be the claim this file exists to prevent. The other three mutations die loudly: reverting the fix is caught by three tests, with the invariant test naming all five truncated labels by page and text; dropping the corridor bound is caught by the served count and deliberately NOT by the invariant test, which asks the opposite question; and minWrapRunes = 0 empties the floor's two named digits. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NL8EwtKTPfkzayvokjzbxV
…tion
This reverses the design two commits back, on the user's instruction after looking at
pages 521 and 522: "maybe just do screenshots, not trying to parse helper text for
images. We can parse it and add as alt for images."
The reasoning is worth keeping because it undoes work. The page had already solved the
layout, and re-laying it out is what produced every defect in this area -- labels
colliding on 522, stranded siblings, orphaned wrapped tails, and each figure placed
blind to its neighbour. All four existed only because the reader was rebuilding an
arrangement the paper prints correctly. Take the arrangement and there is nothing left
to place, to cut, or to collide.
labelBand is the drawing unioned with every run claimLabels reaches, taken whole, with
the top and bottom moved out so no line of type is cut across. EVERY claim, gated or
not, because the two questions have opposite failure modes: a paragraph printed inside
a picture that already prints it is untidy, and a paragraph DESCRIBED to a screen reader
as a diagram's callout is a lie. So figureLabels keeps labelExtent exactly where it was
and now decides only the alt text.
THE PAGE BAND WAS THE FIRST CHOICE AND THE MEASUREMENT REFUSED IT. The cost of a band
is prose the crop prints as pixels that the block flow also emits, counted in runes over
the pages that serve a crop by TestABandCropDuplicatesThisMuchProse:
columns manual sequential manual
the drawing alone, as it was 773 1.0% 51 0.3%
drawing + the gated labels 773 1.0% 116 0.7%
drawing + every claim (SHIPPED) 1,481 1.8% 1,714 9.8%
the printed column 45,736 56.8% 4,355 25.0%
the page's full width 67,662 84.0% 11,377 65.4%
59,618 of the full band's 67,662 runes are a NEIGHBOURING COLUMN's text, and on the
columns manual a neighbouring column is a different language -- the one failure
attribute says the funnel may not have. That document also carries 0 labels, all nine of
its claims being false, so it would pay the whole cost of a feature it cannot use. The
estimate before measuring was "page 521's top band is almost nothing"; it was wrong by
two orders of magnitude, structurally rather than by accident.
It fixes what conversion.md recorded as unfixable. `Кнопка сброса`, `Индикатор Wi-Fi`
and `Датчик края` are printed by the crop -- 45 of the sequential manual's 50 refused
claims are -- and both refusals on the record were about DRAWING them, not cropping.
The fourth name is a correction, and the user photographed it: `Датчики перепада
высоты` is not a refused claim of figure 2, it is claimed by NOTHING, so no gate ever
sees it and the leaders in figure 2's top edge end in empty paper. From figure 0 below
it is 40.44 units away against labelCorridor = 40; from figure 2 above it is within
reach, with a real 3.3x3.3 terminator at (665.3, 339.3) that labelAlign refuses. That
refusal is an asymmetry rather than a threshold: labelAlign is 4 units on both axes, a
one-line label is 13-14 units TALL so +-4 of its midline is most of it, and this label
is 140 units WIDE so +-4 of its midpoint is 6% of it. Asking the mark to fall inside the
run's own extent on that axis is the rule that would claim it, and it is unbuilt here
because it changes claimLabels and every count in this package is expressed in that.
labelCorridor is deliberately not moved to 41 either.
absorbNested comes with it, is a fix rather than a cost, and sits behind ServedFigures
rather than inside FindFigures. Putting it inside was a mistake caught by the pins: it
reshaped the figure set every threshold sweep measures, so TestGrowSweep's label count
fell 229 -> 227 and TestAPlateMergeOnSharedLabelsIsRefused's claim count 327 -> 319 --
both measurements of a pass this one REPLACED, moved by a rule with nothing to do with
either. Ten pinned counts moved and none of them should have. FindFigures is now the
geometry and may return a nested crop; ServedFigures is what a conversion calls and does
not. The only counts that move are what a reader is served. Two drawings that reach the
same run can end up with one band inside the other -- page 529's figure 7 is figure 4's
band with the top cut off, because both claim the numbered step underneath -- and
serving both shows a reader the picture and then its own lower half. The contained crop
is dropped and its labels go to the one that swallowed it: 6 on the sequential manual, 0
on the columns one. It is NOT the plate merge this package refuses, which unions two
want boxes and invents a rectangle larger than either; this invents nothing, the
surviving crop being one that already existed unchanged to the unit, so there is no
cascade to bound. TestNoFigureOverlapsAnotherOnEitherManual keeps nesting at zero
everywhere and turns its overlap half into a per-page census that may only fall.
Measured with `manualbox verify` over the sequential manual in all 34 languages, the
same binary path before and after: blocks 14,749 (1,289 furniture, 165 callout)
IDENTICAL, labels carried 166 identical, median coverage 0.996 both times, reading-order
23 both times, invented-text and the two join kinds identical. Figures 134 -> 128, which
is absorbNested. The one check that moved is figure-clipped, 25 -> 14, because the crop
is wider than the drawing again and further than growth ever reached. Total findings
287 -> 276.
Verified in Chrome on the re-converted demo: page 521 shows every label the paper
prints, page 522's two genuine overlaps are gone, and the columns manual's German is
460 blocks, 53 figures and 0 labels -- the safety property intact.
The costs are stated rather than managed. The band rule produces 35 overlapping crop
pairs on the sequential manual and runs cut by a crop edge go 16 -> 48; what a reader is
served holds 26 pairs, 11 of them on served pages, so page 521's two crops each show a
fragment of the other's label column. The crop can print
text the alt text does not name -- page 522's second figure -- which is the right way
round, a description that says less than the picture shows being a gap and one that says
more a lie. A wrapped label reads as several labels, which is not new but was hidden by
the paper's own arrangement until the strings became the only text. The columns manual's
cover gains a blank band of 23 units, its title being printed well clear of the art.
Figure.Labels is []string. Position and side leave the type, the schema (00009), the
served JSON and the reader; Figure.LabelBoxes hands the run's own coordinates to
Callouts.Mark in-process and is never stored, which retires the fraction-reconstruction
hazard calloutAt documents rather than absorbing it. The order is the page's own, down
then across, because that is the order alt text should read in.
Mutated one at a time: collapsing the band back to the drawing is caught on six pinned
numbers across both manuals, and dropping snapBandToLines is caught by the columns
manual's "runs cut by a crop edge", which goes 0 -> 2.
TWO FIXTURE TESTS WERE ALREADY RED AT 3532d4b and are corrected here with the numbers
they were failing on -- verified by stashing this work and re-running them.
TestAPlateMergeOnSharedLabelsIsRefused wanted 319 claims and 40 on page 521 against the
327 and 43 the wrapped-label tail fix produced, and TestOptingOutIsTodaysConversionExactly
wanted 409 content and 89 callout blocks against 404 and 95. Fixture tests need
MANUALBOX_TEST_FIXTURES=1 and do not run in the default suite, which is how a red test
shipped. The band moves none of those four numbers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V66Tc4bF8tFtgR6zgSzH22
conversion.md keeps the section it supersedes, because the sequence is the argument: grow the crop onto the labels, carry each label as text and let the reader draw it, crop the band and stop re-laying out a page the paper had already laid out. The two claims in the old section that are now false are marked where they stand rather than deleted -- "Reader.tsx draws it beside the picture" and the paragraph on a position that is routinely negative -- because the fact the second turns on, that a label sits outside the drawing, is exactly why the crop had to become a band. Three corrections, each of which was believed and is measured: `Датчики перепада высоты` was recorded as a claim of figure 2 whose side the gate refused. It is claimed by NOTHING: 40.44 units below figure 0 against labelCorridor = 40, and within reach of figure 2 with no terminator pointing at it. 0.44 units from being fixed, which is why the corridor is not moved to 41 -- a bound set from the one sample it has to admit is the fitted threshold this file refuses elsewhere. The estimate for what a page band would duplicate was "page 521's top band is almost nothing; page 522 would be more ... a much smaller problem than the four it removes". It is 84.0% of the columns manual's text and 65.4% of the sequential manual's, and 59,618 of the columns manual's 67,662 duplicated runes are a neighbouring column, which on that document is a different language. A wrapped label reads as several labels, because one entry is one printed LINE. That is not new -- it is claimLabels' own unit -- but it was invisible while the paper's own arrangement put the lines back together, and it is visible now that the strings are the only text a screen reader gets. CLAUDE.md carries the same numbers in short form, and its Russian block sequence gains its fourth entry: 431 -> 449 -> 409 -> 404. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V66Tc4bF8tFtgR6zgSzH22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Twenty-two commits, each building independently. Two real manuals are the acceptance
criteria throughout: a 560-page 34-language manual whose languages run in sequential
sections, and a 68-page 5-language manual whose languages run in parallel columns.
Improving one by altering the other is the regression this branch guards against.
What it does
Create a device, upload a manual, and be told what it contains — for free, before
anything is converted or sent anywhere. It stops at the gate; nothing is spent.
The second half of the branch is about a page that holds more than one language:
internal/doc/runs.goreads where text sits, viapdftohtml -xml. This was amissing prerequisite rather than an enhancement:
DetectColumnsandColumnLanguageshad shipped with nothing able to feed them, so all theirverification was against coordinates typed into a test.
internal/doc/regions.godivides a page on language, never geometry. Columncount is not language count in either direction — one manual sets three languages
on a page, the other sets one language across four table cells.
doc_regionspersists it, keyed on geometry(document_id, source, page, x0),because two German columns on one page collide under any label-based key.
Characters replace pages as the unit of size. Measured: a German-reading household
is charged 44,376 characters of the column manual instead of the 233,849 its pages
hold in all five languages — 19%, where before one language of five cost the same as
all five.
Verified
make checkgreen; both fixtures pass withMANUALBOX_TEST_FIXTURES=1five languages read back from SQLite, and the eight human-verified pages matching
column for column
34-section language map unchanged
00003rebuilds two shipped tables; every row is asserted identicalacross it, and the rebuild's atomicity under
foreign_keys(1)was measured ratherthan assumed
Four claims that measurement killed
Each was believed, then disproved, and the code now says what is true:
costs 1 column of 169. Only attribution collapses, 53 tag-named columns to 0.
Rendering two at 108 dpi showed they are side-by-side tables and the detector was
right.
it is 93% accurate at every threshold from 1 to 50 marks, one error carrying 118.
Known and deliberately not fixed here
Regions are stored but not surfaced.
ingest.Gatestill prices from theper-page
doc_langsrows, so the column manual currently summarises for a user as"68 pages in 2 languages, none of them yours. It has fax and Ukrainian." — while
doc_regionsholds all five languages correctly.CostEstimate.Charsis documentedas always present and has never been set.
The cause of both is that the printed-index parser cannot read a contents page laid
out in columns: it yields the vocabulary
[FAX GA NDE UA VIA Z], soFAXbecomes alanguage and a single-letter
Dtab loses its corroboration. That also explains theone language error in the column manual — a German table cell read as Finnish,
because its printed
Dwas rejected and onlyäandöwere left to judge by.Surfacing and the parser fix are the next slice.
🤖 Generated with Claude Code
https://claude.ai/code/session_01U5ihvamdb4fsiMr3zoafPJ