Skip to content

Fix bilingual dictation translating/duplicating output; correct provider request fields - #16

Merged
xarthurx merged 13 commits into
mainfrom
fix/bilingual-prompt-and-provider-requests
Oct 13, 2026
Merged

xarthurx merged 13 commits into
mainfrom
fix/bilingual-prompt-and-provider-requests

Conversation

@xarthurx

Copy link
Copy Markdown
Owner

What changed

Root cause — Chinese dictation mixed with English (bilingual mode). The bilingual Whisper prompt was a Chinese sentence followed by its English translation. Whisper treats the prompt as the preceding transcript and imitates it, writing every spoken segment twice: as an interleaved English translation (那你就记得。 Then you remember.) or a verbatim duplicate (Thank you for the email. Thank you for the email.). Bilingual mode now conditions on one language's sentence (first slot that has one), with the dictionary after a Latin-script sentence and before a CJK one.

Enhancement. Bilingual cleanup uses a code-switch-preserving rule (never translate; Chinese / Japanese / English punctuation each in its own convention). Bare base codes no longer fall into the forced "respond in X regardless of what the user spoke" template: zh → its tailored instruction; en (en-US/en-GB) → the language-preserving auto instruction, so no spelling convention is imposed.

Provider requests (audit against current provider docs):

  • verbose_json only for Whisper models — gpt-transcribe rejects it (HTTP 400).
  • gpt-transcribe: languages[] (both languages in bilingual mode) + sanitized keywords[]; detection read leniently from languages[].code.
  • gpt-4o-transcribe-diarize: no prompt.
  • Mistral: dictionary as context_bias, no prompt/response_format; voxtral-small-latest (not a transcription-endpoint model) removed and migrated to voxtral-mini-latest.
  • Refactor: request closures take RequestHints { prompt, vocabulary }; the dictionary-echo retry drops both.

Normalization. CJK small-form marks (﹐﹑﹒﹔﹕﹖﹗) → standard full-width; in a pair that includes Chinese, Han text under an English resolution keeps Chinese punctuation + Simplified (other pairs such as ja+en keep plain English handling, so kanji are never simplified); more Chinese silence-hallucination phrases.

Evidence

  • History DB: 0 interleaved translations in ~1,000 Chinese dictations Feb–May 2026; the first appeared 2026-06-04, the day bilingual mode (v0.8.0) shipped.
  • 22-clip eval on Groq whisper-large-v3 (TTS + noise/pauses; Chinese, English, code-switched): old prompt 12/22 clean (10 duplicated/translated/echoed); the shipped shapes 22/22.
  • Live run of the real Rust request path (transcribe_groq + new prompt builder) against Groq: fields accepted, detection returned, no translation/duplication.

Verification

  • cargo test: 255 passed · bun test: 30 passed · bun run typecheck clean · bun run build OK
  • cargo clippy --all-targets: 8/9/1 warnings vs baseline 12/13/1 (none in changed code)
  • Fresh whole-branch review: 3 Important findings + 1 re-graded Minor fixed, each with a failing-first regression test.

Not verified live

  • gpt-transcribe request shape (dev machine's OpenAI key returns 401) and Mistral context_bias (no key). Both follow the providers' docs/SDK encoding; detection parsing is lenient so a shape surprise only loses detection. The Standard default stays gpt-4o-mini-transcribe until a live check (TODO).

Plan: docs/superpowers/plans/2026-10-13-bilingual-prompt-and-provider-requests.md

🤖 Generated with Claude Code

Copilot AI balanced review requested due to automatic review settings October 13, 2026 10:19

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Pair-aware cleanup normalization is incorrectly applied to chat responses and can rewrite valid non-Chinese Han text.

2 open findings
What changed in this PR

Fixes bilingual transcription duplication/translation and aligns provider request payloads with model-specific APIs.

Changes:

  • Uses single-language bilingual conditioning and language-preserving cleanup.
  • Adds provider-specific vocabulary, language, and response handling.
  • Improves CJK normalization, hallucination filtering, migrations, tests, and documentation.
File Description
tests/​retiredModels.test.ts Tests Mistral model migration.
tests/​prompts.test.ts Tests bilingual language instructions.
src/​utils/​languageSupport.ts Adds bilingual and base-code handling.
src/​services/​tauriApi.ts Passes language pairs to reasoning.
src/​models/​modelRegistryData.json Removes unsupported Voxtral Small.
src/​hooks/​useTranscriptionPipeline.ts Threads bilingual context through cleanup.
src/​config/​retiredModels.ts Migrates retired Mistral model.
src/​config/​prompts.ts Applies bilingual cleanup prompts.
src-tauri/​src/​transcription/​normalize.rs Adds pair-aware CJK normalization.
src-tauri/​src/​transcription/​mod.rs Revises bilingual Whisper conditioning.
src-tauri/​src/​transcription/​hallucination.rs Expands Chinese hallucination filtering.
src-tauri/​src/​transcription/​dictionary_echo.rs Separates prompt and vocabulary hints.
src-tauri/​src/​transcription/​cloud.rs Adds provider-specific request fields.
src-tauri/​src/​commands/​transcription.rs Threads languages and request hints.
src-tauri/​src/​commands/​reasoning.rs Applies pair-aware response normalization.
docs/​TODO.md Records pending live verification.
docs/​superpowers/​plans/​2026-10-13-bilingual-prompt-and-provider-requests.md Documents implementation plan.
docs/​PROGRESS.md Records diagnosis and evaluation results.
docs/​CHANGELOG.md Documents user-visible fixes.
docs/​ARCHITECTURE.md Updates transcription architecture.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

undefined,
temperature,
resolvedLanguage,
languagePair,
Comment thread docs/ARCHITECTURE.md
invoke("process_reasoning", { text, provider, model, system_prompt, api_key })
→ Context-aware dictionary aliases are supplied as explicit correction mappings
→ Bilingual mode uses a code-switch-preserving language rule (never translate);
other modes use the registry instruction (bare codes like `en` map to `en-US`)
@xarthurx
xarthurx merged commit 69cef7f into main Oct 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants