From 2a9abbc5aaf228fe232fe52074dd9e468d4d9102 Mon Sep 17 00:00:00 2001 From: Chris Thompson Date: Tue, 22 Sep 2026 11:55:19 -0600 Subject: [PATCH 1/2] chore: bump vernacula-phonemizer to #1420 (a rewrite must not collapse provenance under it) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pin ac60b3c4 -> 6e2165c7, five commits. The one this repo was waiting for is #1420: normalizeRomans rewrites on \p{L}+ and runs over every language, and in a script without spaces there is no word break for that match to stop at, so it matched the whole clause and stamped its span across every token it produced. Measured on the raw trace rather than on our own units, because the local merge collapses these downstream either way and counting units would have measured the workaround instead of the fix: ac60b3c4 6e2165c7 ja rows with 2+ tokens sharing a span 14 3 cmn rows with 2+ tokens sharing a span 3 3 ja 14 -> 3 reproduces upstream's figure exactly. MANDARIN WAS NEVER AFFECTED, and the reason is worth keeping: the cmn trace returns ONE token per sentence carrying per-syllable IPA, so a whole-clause match has no second token to flatten against. The defect needed several tokens to collapse and only ja had them. The cmn residue of 3 was never this bug. What the reader gets, through KokoroPhonemizer on the same corpora: median chars per unit median units per sentence ja 4.33 -> 4.08 11 -> 12 cmn 1.00 -> 1.00 35 -> 35 Eleven rows stop being merged into coarser units and get their real boundaries back. Small in the median because it is 11 of 123 rows -- but those were the worst rows, the ones offering eight units each covering 30 of 34 characters. The merge added in #240 is NOT inert afterwards: ja keeps 3 rows of numeral/unit overlap ("83 m" against "83 mです"), which is overlap rather than collapse. Run 4 predicted the guard would stop firing; Run 6 corrected that and this measures it. Cold-trace gate re-run on the new pin: 189 of 189 languages, no poisons. Suite: 95 + 257 + 23 + 362, 0 failures. Scope: one commit is core provenance (Rewriter, Provenance, JsRegex and the TypeScript twin); the rest are English data and lexicon work, including #1418 reverting 26 rows on re-arbitrated evidence and #1425 porting code-slot unit guards to C#. So this carries English behaviour changes as well as the fix. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01KfTCTjC7FTnNm3aS2zszKE --- ...a_word_level_segmentation_investigation.md | 47 +++++++++++++++++++ external/vernacula-phonemizer | 2 +- 2 files changed, 48 insertions(+), 1 deletion(-) diff --git a/docs/investigations/ja_word_level_segmentation_investigation.md b/docs/investigations/ja_word_level_segmentation_investigation.md index 02f99a7..bba53d5 100644 --- a/docs/investigations/ja_word_level_segmentation_investigation.md +++ b/docs/investigations/ja_word_level_segmentation_investigation.md @@ -312,3 +312,50 @@ token means a subdivision would have no measured audio boundary behind it, so ex would hand this repo an estimate dressed as a boundary. If the audio side ever gains per-mora groups the question reopens; until then the answer to "can Japanese highlighting be more word-level" is that it already is, and the thing that made it feel otherwise is fixed at both ends. + +## Run 7 — 2026-09-22 — taking the upstream fix, and what it actually bought + +Pin `ac60b3c4` → `6e2165c7`, five commits. Measured rather than relayed, because the interesting +number is the one this repo sees rather than the one upstream reports. + +**The upstream defect, counted on the raw trace** — the local merge collapses these downstream +either way, so counting units would have measured our own workaround instead of their fix: + +``` + ac60b3c4 6e2165c7 (#1420) +ja rows with 2+ tokens sharing a span 14 3 +cmn rows with 2+ tokens sharing a span 3 3 +``` + +`ja` 14 → 3 reproduces their figure exactly. + +⚠ **Mandarin was never affected, and the reason is worth keeping.** The `cmn` trace returns ONE +token per sentence carrying per-syllable IPA, so a whole-clause `\p{L}+` match has no second token +to flatten against. The defect needed several tokens to collapse and only `ja` had them — which also +means the `cmn` residue of 3 was never this bug and is unchanged by its fix. + +**What the reader gets**, measured through `KokoroPhonemizer` on the same corpora: + +``` + median chars per unit median units per sentence +ja 4.33 -> 4.08 11 -> 12 +cmn 1.00 -> 1.00 35 -> 35 +degenerate / overlapping / out-of-range: 0 throughout +``` + +Eleven rows stop being merged into coarser units and get their real boundaries back. Small in the +median because it is 11 of 123 rows, and those rows were the worst ones. + +**The merge is not inert and the residue confirms it.** `ja` keeps 3 rows of numeral/unit overlap +(`83 m` against `83 mです`), which is overlap rather than collapse. Run 4 predicted the guard would +stop firing once upstream landed; Run 6 corrected that, and this measures it: still 3 rows relying +on it. + +Cold-trace gate re-run on the new pin: **189 of 189 languages, no poisons**. Full suite: +95 + 257 + 23 + 362, 0 failures. + +**Scope.** Of the five commits, one is core provenance (`Rewriter`, `Provenance`, `JsRegex`, and the +TypeScript twin) and the rest are English data and lexicon work, including #1418 reverting 26 rows on +re-arbitrated evidence and #1425 porting code-slot unit guards to C#. So this carries English +behaviour changes as well as the fix — the same shape as the previous bump, and the reason the +English goldens are worth a look rather than a nod. diff --git a/external/vernacula-phonemizer b/external/vernacula-phonemizer index ac60b3c..6e2165c 160000 --- a/external/vernacula-phonemizer +++ b/external/vernacula-phonemizer @@ -1 +1 @@ -Subproject commit ac60b3c464ae769b6b4054788b1cc1ab36b99bb8 +Subproject commit 6e2165c7f5221de141b221b87a3cfa0a0050406b From 984d4e80c3ee1efd02df75e6193597eae17ba46f Mon Sep 17 00:00:00 2001 From: Chris Thompson Date: Tue, 22 Sep 2026 11:57:52 -0600 Subject: [PATCH 2/2] review: measure the claim about English rather than warning about it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run 7 said the bump "carries English behaviour changes as well as the fix", which is true of the dictionary and vaguer than the evidence supports. The part the review actually had to check is that #1420 changes core provenance for EVERY language while our trace coverage outside ja/cmn is thin. Measured across both pins: en/es/fr token counts identical, InputSpan coverage 100% on both, and English rows with shared spans 25 -> 23. No spans lost, no new collapses. The residual shared-span rows are the legitimate kind -- a numeral expansion producing several tokens from one source span -- and English does not reach FromTrace at all, so they do not affect units here either way. On readings: 0 of 114 English golden rows differ between the pins. #1418 and #1425 target specific lexical items (µin, CoCr, V6L 2T5) the golden prose corpus does not contain. ZERO-DIFF MEANS "NOT EXERCISED", NOT "NO CHANGE". Those rows genuinely changed. The honest statement is narrower than the warning and narrower than a clean bill of health: real English changes, aimed at tokens absent from our measurable surface, not detectable from this repo. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01KfTCTjC7FTnNm3aS2zszKE --- ...a_word_level_segmentation_investigation.md | 39 ++++++++++++++++--- 1 file changed, 34 insertions(+), 5 deletions(-) diff --git a/docs/investigations/ja_word_level_segmentation_investigation.md b/docs/investigations/ja_word_level_segmentation_investigation.md index bba53d5..df6f57c 100644 --- a/docs/investigations/ja_word_level_segmentation_investigation.md +++ b/docs/investigations/ja_word_level_segmentation_investigation.md @@ -354,8 +354,37 @@ on it. Cold-trace gate re-run on the new pin: **189 of 189 languages, no poisons**. Full suite: 95 + 257 + 23 + 362, 0 failures. -**Scope.** Of the five commits, one is core provenance (`Rewriter`, `Provenance`, `JsRegex`, and the -TypeScript twin) and the rest are English data and lexicon work, including #1418 reverting 26 rows on -re-arbitrated evidence and #1425 porting code-slot unit guards to C#. So this carries English -behaviour changes as well as the fix — the same shape as the previous bump, and the reason the -English goldens are worth a look rather than a nod. +**Scope, and the part the review actually had to check.** Of the five commits, one is core +provenance (`Rewriter`, `Provenance`, `JsRegex`, and the TypeScript twin) and the rest are English +data and lexicon work, including #1418 reverting 26 rows on re-arbitrated evidence and #1425 porting +code-slot unit guards to C#. + +⚠ **The core change touches every language, and our trace coverage outside `ja`/`cmn` is thin** — a +regression there would show as lost spans or new collapses in languages nothing here asserts on. +Measured across both pins: + +``` + tokens withInputSpan rows with shared spans +en ac60b3c4 2593 2593 (100%) 25 +en 6e2165c7 2593 2593 (100%) 23 +es / fr unchanged on every column +``` + +Token counts identical, span coverage 100% on both, English shared-span rows 25 → 23. No spans lost +and no new collapses. The residual shared-span rows in `en`/`es`/`fr` are the legitimate kind — a +numeral expansion producing several tokens from one source span — and English does not reach +`FromTrace` at all, so they do not affect units here either way. + +On the English *readings*, the first draft of this entry said the bump "carries English behaviour +changes", which was vaguer than the evidence supports. Kokoro phonemes for every English golden row, +both pins: + +``` +English golden rows whose reading differs: 0 of 114 +``` + +⚠ **Zero-diff means "not exercised", not "no change".** Those dictionary rows genuinely changed; +#1418 and #1425 target specific lexical items (`µin`, `CoCr`, `V6L 2T5`) that the golden prose +corpus does not contain. So the honest statement is narrower than the warning *and* narrower than a +clean bill of health: the English changes are real, aimed at tokens absent from our measurable +surface, and not detectable from this repo.