Skip to content

feat(voice): add Qwen3-TTS voice design, consented cloning, and fine-tuning (#5381) - #5402

Merged
atomantic merged 6 commits into
mainfrom
claim/issue-5381
Aug 29, 2026
Merged

feat(voice): add Qwen3-TTS voice design, consented cloning, and fine-tuning (#5381)#5402
atomantic merged 6 commits into
mainfrom
claim/issue-5381

Conversation

@atomantic

Copy link
Copy Markdown
Owner

Closes #5381

Summary

Adds Qwen3-TTS local voice backend integration across the voice engine contract, voice profiles, and Universe Voice Lab:

  • Isolated Python Runtime & Probing: Adds health probing, hardware detection (CUDA / Apple Silicon MPS / CPU), model management, and explicit on-demand downloads (server/services/voice/qwen3TtsRuntime.js, scripts/qwen3_tts_runner.py). No boot-time cold inference or downloads.
  • Voice Design: Instruction- and seed-controlled candidate voice generation via Qwen3-TTS 1.7B Voice Design model.
  • Consented Single-Speaker Cloning: Rapid cloning with machine-local reference audio, SHA256 integrity hashing, exact transcript conditioning, and required performer consent confirmation. Reference audio remains strictly machine-local.
  • Voice Fine-Tuning: Optional, cancellable, checkpointed fine-tuning lifecycle. Emits intermediate checkpoints with audition samples and supports explicit promotion of any selected checkpoint.
  • Interactive Routing Qualification: Evaluates first-audio latency benchmark on the local host before enabling the interactive route (studio remains separately gated).
  • Voice Lab UI: Adds tabbed Voice Lab interface in CharacterDetailEditor for preset promotion, voice design, consented cloning, fine-tuning management, and benchmark audition.

Key Changes

  • scripts/qwen3_tts_runner.py: Standalone CLI runner for runtime probing, voice design inference, consented cloning, standard synthesis, and step-based fine-tuning.
  • server/services/voice/qwen3TtsRuntime.js: Probing, GPU/hardware capability inspection, model catalog, and user-initiated downloads.
  • server/services/voice/tts-qwen3.js: Qwen3-TTS synthesis adapter handling prompt conditioning, reference alignment, and rate controls.
  • server/services/voice/fineTuning.js: Machine-local dataset validation, background training job runner, progress tracking, and checkpoint promotion.
  • server/services/voice/profiles.js: Support for designed, cloned, and fine-tuned profile kinds with explicit candidate drafting and approved promotion.
  • server/services/voice/profileBenchmarks.js: Latency benchmark qualification for interactive speech.
  • server/routes/voice.js: REST endpoints for design, clone, promotion, fine-tuning lifecycle, and Qwen3 status/downloads.
  • client/src/components/universe/CharacterDetailEditor.jsx: Full Voice Lab UI supporting design previews, consent verification, fine-tuning controls, and candidate audition.
  • client/src/services/apiVoice.js: Extended API wrappers for all new voice endpoints.

Testing

  • server/services/voice/qwen3TtsRuntime.test.js: Validates model registry, health probing, and download requests.
  • server/services/voice/tts-qwen3.test.js: Validates voice design synthesis and effective controls.
  • server/services/voice/profiles.test.js: Validates draft candidate creation, mandatory consent enforcement, and checkpoint promotion.
  • server/services/voice/fineTuning.test.js: Validates dataset verification, training progress, checkpoint generation, and job cancellation.
  • server/routes/voice.test.js: Validates all new REST endpoints and upload cap enforcement.
  • client/src/components/universe/CharacterDetailEditor.test.jsx: Validates Voice Lab tabs, consent gating, and interactive qualification.
  • Full client and server test suites run cleanly.

@atomantic
atomantic merged commit 739566a into main Aug 29, 2026
7 checks passed
@atomantic
atomantic deleted the claim/issue-5381 branch August 29, 2026 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Qwen3-TTS voice design, consented cloning, and optional fine-tuning

1 participant