feat(accent): local UniDic + POS-driven patches

[torrid-fish]

目的

Closes #45, Closes #48, Closes #50, Closes #57, Closes #58.
Supersedes (now-closed) PRs #47 and #49.
Replaces #51 (same branch, renamed spike/local-unidic → feat/local-unidic;
the rename closed #51, so this is its continuation).

Issue #50 was explicitly framed as an alternative architecture to #48 ("two architectures targeting the same goal. Production will likely pick ONE"). This PR is the GO outcome: local fugashi + NINJAL UniDic CWJ 2025-12-31 in-process replaces the Yahoo MA HTTP round-trip, while keeping OJAD for surface-level phrase pitch and keeping every POS-driven rule originally introduced in PR #49.

This is the single, complete migration PR — it targets main directly and folds in the foundation that was previously proposed as the standalone PR #47 (regex reading-override layer, Needleman-Wunsch DP aligner, /MarkAccent/stream/ NDJSON endpoint). Since the Yahoo-backed intermediate from #47 is a stepping stone that won't ship independently, #47 was closed and its commits are the base of this branch rather than a separate merge.

PR structure (19 commits, base ci/cd-ghcr → main)

Stacked on top of PR #55 (ci/cd-ghcr → main), which carries the generic CI/CD basework (GHCR workflow, Node 24 bumps, non-root Dockerfile). Once #55 merges, this PR's base auto-retargets back to main and the diff stays the same — only the foundation + UniDic + UniDic-specific Docker tweak remain here.

ci/cd-ghcr → foundation (3) → UniDic migration (15) → UniDic Docker tweak (1):

Foundation (was #47): Needleman-Wunsch DP aligner (+ rendaku fold), regex reading-override layer + URL/non-JP preprocessing, and the /MarkAccent/stream/ NDJSON endpoint.

UniDic migration (#50): swap Yahoo MA → fugashi+UniDic, drop Yahoo config, add tokenizer / preprocess / postprocess / user_patches modules, UniDic strong-mode fields, English README.

UniDic Docker tweak: scripts/download_unidic.sh at build time (depends on the unidic pip dep that lands in this PR — that's why it can't live in #55).

Rebased onto current main (which includes the #52 package refactor and the #46 Docker compose setup). The Docker-scaffolding commit from the original spike was dropped during rebase — already upstream via #46. Commit subjects were reworded to stay under 75 chars (so exact hashes aren't pinned here).

Architecture

flowchart LR
    text([text input])
    fugashi["fugashi (MeCab) + UniDic CWJ 2025-12-31<br/>surface · kana · lemma<br/>pos · conjugation · aType"]
    ojad["OJAD scrape (suzukikun)<br/>per-mora surface pitch contour"]
    align["DP alignment<br/><code>align_accent</code>"]
    patches["<code>apply_accent_patches</code><br/>POS-driven rules"]
    out([WordAccentResult])

    text --> fugashi
    text --> ojad
    fugashi -- tokens + POS --> align
    ojad -- per-mora pitch --> align
    align --> patches
    fugashi -. POS metadata .-> patches
    patches --> out

Loading

Simplified schematic. For the full pipeline data-flow diagram see api/accent/README.md → "Data flow".

Yahoo MA HTTP endpoint fully removed (api/accent/furigana.py and the /MarkFurigana/ route deleted). Everything #49 introduced (POS-driven apply_accent_patches, pos_match override predicate, 5 POS columns on WordResult/WordAccentResult) carries over to UniDic identically — same UniDic schema, only the upstream changed.

Package layout

• fugashi + UniDic tokenizer → api/accent/tokenizer.py
• preprocessing (URL / comma / × strip + restore) → api/accent/preprocess.py
• postprocessing passes (heiban-particle flatten, 助詞 furigana suppression, punct suppression, katakana toggle) → api/accent/postprocess.py
• POS-driven patches + user-maintained patch table → api/accent/reading_overrides.py + api/accent/user_patches.py
• strong-mode fields → api/accent/models.py
• chunked orchestrator (_build_chunks, _schedule_chunks) → api/accent/pipeline.py

Local UniDic replace Yahoo MA

• _fetch_yahoo_raw → _tag_local using singleton fugashi.Tagger() + UniDic CWJ 2025-12-31
• Field mapping: feat.pos1 → pos, feat.pos2 → pos1, feat.cType → conjugation_type, feat.cForm → conjugation_form, feat.lemma → base (with -gloss suffix stripped), feat.aType → lexical_kernel (parsed for multi-reading via _parse_atype)
• Reading uses feat.kana over feat.pron — UniDic stores 忙しい as kana=イソガシイ (matches OJAD's ortho-kana) while pron=イソガシー with chōonpu would never align

Strong-mode fields

• lexical_kernel: int | None — aType primary (0=heiban, N≥1=kernel on mora N)
• lexical_kernel_alts: list[int] | None — multi-reading alternates (e.g. aType="2,0" → [2, 0])
• kernel_absorbed: bool — UniDic says kernel exists but OJAD has no FALL in this word's range (connected-speech sandhi case, e.g. 忙しい inside お忙しい中)

Dictionary variant selection

NINJAL publishes two UniDic variants — the download script supports both:

Tag	Variant	Training corpus	Use case
cwj-2025-12-31	CWJ (現代書き言葉)	BCCWJ (written)	Articles, novels, web text — default
csj-2025-12-31	CSJ (現代話し言葉)	CEJC (spoken)	Conversational / transcribed speech

./scripts/download_unidic.sh cwj-2025-12-31   # default — written
./scripts/download_unidic.sh csj-2025-12-31   # spoken-language variant
./scripts/download_unidic.sh cwj-2021-08-31   # older CWJ 3.1.0

CWJ is the default because this service primarily processes written input. CSJ may be preferable when processing conversational or transcribed speech. The Dockerfile defaults to CWJ; override by changing the script argument in the builder stage.

Hide internal MA metadata from response

base/pos/pos1/conjugation_type/conjugation_form marked Field(exclude=True) — kept on the model for in-pipeline use by apply_accent_patches, excluded from JSON serialization. Cleaner client contract; nothing functional changed.

Unify chunking between two endpoints

_build_chunks / _schedule_chunks shared by /MarkAccent/ and /MarkAccent/stream/. Both endpoints emit byte-identical per-chunk word lists; only delivery shape differs (collected vs streamed NDJSON).

Rendering polish

What	Why
Polish furigana/accent on punct, symbols, toggles	Existing edge cases pre-spike
Katakana with render_katakana_furigana=False keeps accent (only clears furigana)	Per-mora pitch overlay can still render against the surface text
助詞 (pos=助詞) tokens clear their redundant top-level furigana	hiragana ruby on hiragana surface is visual duplication; was crowding out pitch overlay
After heiban predecessor, の／な／は／が pitch flattened to LOW	OJAD's HIGH-plateau continuation was visual noise without contour change
\d × \d swapped → \d/\d pre-pipeline, × restored on surface post-alignment	OJAD merges 19×19 → 1919 (千九百十九); swap forces independent reading per number
Numeric _match_cost adds tiny per-empty-OJAD-entry cost	DP was splitting 19/19 as 1+7 morae instead of 4+4 because all cost-0 ties picked an arbitrary path
User-maintained patch table (user_patches.py) with explicit per-mora int tuples	Hand-fix OJAD/UniDic misreads without touching pipeline code

Verified locally

• ruff check, ruff format --check pass (17 files); import main + all accent submodules import clean
• 学校に行きます。今日は寒いですね。19×19の格子。 — heiban-particle flatten (に stays 1, は／が／の／な flatten after heiban), 助詞 furigana cleared, 19×19 splits 4+0+4 not 1+7
• Response JSON contains zero of base/pos/pos1/conjugation_type/conjugation_form (Field(exclude=True))
• /MarkAccent/stream/ returns NDJSON with {chunk, subchunk, status, result, error}

Disambiguation probe

All 11 cases from #49's table still pass — POS gates fire identically because the UniDic schema is the same upstream of the rule layer.

UniDic Docker tweak (the only Docker/CI commit still in this PR)

• UniDic dict baked into the image: the unidic pip package ships only the loader, not the ~1.3GB dicdir, so fugashi.Tagger() would fail at runtime without the dictionary. Dockerfile now runs scripts/download_unidic.sh in the builder stage as its own cache layer → the image is self-contained, no runtime download.

The CD workflow, Node 24 bumps, and non-root runtime are in #55 (the base of this stacked PR).

The published CD image after both #55 and this PR is ~2.1GB (base + venv + UniDic CWJ 2025-12-31). Smoke-tested end-to-end before the split: build + push to GHCR green, image runs as uid 10001, and POST /api/MarkAccent/ returns 200 with correct accent output.

Out of scope

• i-adjective た形 cross-token (高かった = 高 + かっ + た)
• Loanword 3-mora rule (バナナ, ピアノ)
• 数詞 + counter irregulars beyond date/age (4時, 7時, 9時, 1分, 1人, 2人)
• Replacing OJAD entirely — still needed for surface-level phrase pitch; UniDic aType is lexical (dictionary-form), not connected-speech contour

Notes

• Docker image disk cost: UniDic CWJ 2025-12-31 ≈ 1.3GB on disk. Trade-off accepted per spike's Evaluate local UniDic (fugashi/Sudachi) for in-process accent + lower latency #50 measurement.
• Singleton tagger: per-request construction would re-load ~1.3GB and add ~1-2s; tagger is cached as module-level lazy init.

🤖 Generated with Claude Code

[github-actions]

🛡️ PR Quality Check Summary

✅ PR Title: Passed (Length: 47/75, Format: OK). feat(accent): local UniDic + POS-driven patches
✅ Branch Name: Follows naming convention (feat/local-unidic)
✅ Commit Messages: All 29 commit(s) passed (Length, Format, Case)
✅ Conflicts: No merge conflict markers found
✅ Python Quality: All checks passed.

---

🎉 All checks passed!

[wade00754]

感謝更新，前一輪提到的 streaming script、pending task cancellation、部分 logging 問題都有改善。不過這裡還有兩個 OJAD fallback 相關問題，建議 merge 前修掉。

1. 🟡 [important] OJAD HTML 結構異常時會靜默退化，但 response 沒有 warning

api/accent/ojad.py:79-87 目前如果 OJAD 回傳 HTTP 200，但 HTML 結構改變、維護頁、rate limit 頁，導致找不到 div.phrasing_text，只會 log warning，然後回傳空的 ojad_results。

後續 api/accent/pipeline.py:153-162 只有在 OJADUnavailableError 時才會設定 warning。因此這種情況會變成 status=200、furigana-only fallback，但 response 沒有任何 warning，client 會以為 pitch accent 是正常結果。

建議：當 phrasing_texts 為空，或 phrasing_texts / phrasing_subscripts 數量不一致時，在 get_ojad_result() raise OJADUnavailableError，讓 pipeline 走同一個 graceful degradation 路徑並帶 warning。

2. 🟡 [important] OJAD markup 局部變形可能變成 hard 500，而不是 graceful degradation

api/accent/ojad.py:97-105 直接假設每個 span 都有 class：

if moji["class"][0] == "accent_plain":

如果 OJAD HTML 有局部變動，moji["class"] 可能丟 KeyError。這不會被 OJADUnavailableError 捕捉，最後會進到 pipeline.py:199-207 的 generic exception，回傳 500。

這和目前新增的「OJAD 不可用時退化成 furigana-only」設計目標不一致。

建議：在 get_ojad_result() 中把 HTML parse error（例如 KeyError / IndexError / AttributeError）轉成 OJADUnavailableError，或改用 defensive parsing，讓 malformed OJAD response 也走 graceful degradation。

[torrid-fish]

Closing in favour of PR #63 (commercializable OpenJTalk-only backend). OJAD cannot be used for commercial deployment, so this OJAD/UniDic fallback line is abandoned. The non-OJAD work (local UniDic, accent package split, alignment, USER_PATCHES, per-mora furigana, etc.) carries forward through #63's stack which now targets main directly.