Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions docs/investigations/ja_word_level_segmentation_investigation.md
Original file line number Diff line number Diff line change
Expand Up @@ -312,3 +312,79 @@ token means a subdivision would have no measured audio boundary behind it, so ex
would hand this repo an estimate dressed as a boundary. If the audio side ever gains per-mora groups
the question reopens; until then the answer to "can Japanese highlighting be more word-level" is that
it already is, and the thing that made it feel otherwise is fixed at both ends.

## Run 7 — 2026-09-22 — taking the upstream fix, and what it actually bought

Pin `ac60b3c4` → `6e2165c7`, five commits. Measured rather than relayed, because the interesting
number is the one this repo sees rather than the one upstream reports.

**The upstream defect, counted on the raw trace** — the local merge collapses these downstream
either way, so counting units would have measured our own workaround instead of their fix:

```
ac60b3c4 6e2165c7 (#1420)
ja rows with 2+ tokens sharing a span 14 3
cmn rows with 2+ tokens sharing a span 3 3
```

`ja` 14 → 3 reproduces their figure exactly.

⚠ **Mandarin was never affected, and the reason is worth keeping.** The `cmn` trace returns ONE
token per sentence carrying per-syllable IPA, so a whole-clause `\p{L}+` match has no second token
to flatten against. The defect needed several tokens to collapse and only `ja` had them — which also
means the `cmn` residue of 3 was never this bug and is unchanged by its fix.

**What the reader gets**, measured through `KokoroPhonemizer` on the same corpora:

```
median chars per unit median units per sentence
ja 4.33 -> 4.08 11 -> 12
cmn 1.00 -> 1.00 35 -> 35
degenerate / overlapping / out-of-range: 0 throughout
```

Eleven rows stop being merged into coarser units and get their real boundaries back. Small in the
median because it is 11 of 123 rows, and those rows were the worst ones.

**The merge is not inert and the residue confirms it.** `ja` keeps 3 rows of numeral/unit overlap
(`83 m` against `83 mです`), which is overlap rather than collapse. Run 4 predicted the guard would
stop firing once upstream landed; Run 6 corrected that, and this measures it: still 3 rows relying
on it.

Cold-trace gate re-run on the new pin: **189 of 189 languages, no poisons**. Full suite:
95 + 257 + 23 + 362, 0 failures.

**Scope, and the part the review actually had to check.** Of the five commits, one is core
provenance (`Rewriter`, `Provenance`, `JsRegex`, and the TypeScript twin) and the rest are English
data and lexicon work, including #1418 reverting 26 rows on re-arbitrated evidence and #1425 porting
code-slot unit guards to C#.

⚠ **The core change touches every language, and our trace coverage outside `ja`/`cmn` is thin** — a
regression there would show as lost spans or new collapses in languages nothing here asserts on.
Measured across both pins:

```
tokens withInputSpan rows with shared spans
en ac60b3c4 2593 2593 (100%) 25
en 6e2165c7 2593 2593 (100%) 23
es / fr unchanged on every column
```

Token counts identical, span coverage 100% on both, English shared-span rows 25 → 23. No spans lost
and no new collapses. The residual shared-span rows in `en`/`es`/`fr` are the legitimate kind — a
numeral expansion producing several tokens from one source span — and English does not reach
`FromTrace` at all, so they do not affect units here either way.

On the English *readings*, the first draft of this entry said the bump "carries English behaviour
changes", which was vaguer than the evidence supports. Kokoro phonemes for every English golden row,
both pins:

```
English golden rows whose reading differs: 0 of 114
```

⚠ **Zero-diff means "not exercised", not "no change".** Those dictionary rows genuinely changed;
#1418 and #1425 target specific lexical items (`µin`, `CoCr`, `V6L 2T5`) that the golden prose
corpus does not contain. So the honest statement is narrower than the warning *and* narrower than a
clean bill of health: the English changes are real, aimed at tokens absent from our measurable
surface, and not detectable from this repo.
Loading