AI auto scripture: local transcription + AI reference detection (opt-in)

[josephomills]

Builds on #3578 (its commit is included here) — happy to rebase once that lands, or fold both together if preferred.

What

AI auto scripture (opt-in, off by default): FreeShow listens to the sermon, transcribes locally with whisper.cpp, detects scripture references — both spoken ("John chapter 3 verse 16", instant local matching) and quoted ("for God so loved the world…", via a bring-your-own-key LLM) — and either projects them automatically or offers one-click suggestion cards in the scripture drawer.

https://github.com/ChurchApps/FreeShow — demo video to follow.

Features

• Live panel in the Scripture drawer: status, live transcript ticker (expandable), suggestion cards with confidence badges, restore-previous, manual-override auto-pause
• Fully local option: a Local (Ollama) provider — Gemma / Llama / Qwen / Mistral / Phi on the user's own machine, no key, nothing leaves the device
• Two-tier detection: instant local matching for explicit references (works with zero network/keys) + optional LLM detection of quoted/paraphrased verses (Anthropic / OpenAI / Gemini via plain REST, user's own key)
• Quality gates: confidence-gated auto-projection (quoted verses default to suggestions), quoted matches verified against actual verse text across the user's selected translations, dedupe/cooldowns, music segments (♪) excluded from detection
• Voice commands (separate opt-in): "go to the next verse", "give me verse five", "give me NIV", "give me another translation" — imperative-only, and only while a scripture is live
• Interpretation mode: per-window language auto-detect for services with live interpreters, constrained to declared languages, with a "detect scriptures from" selector; multi-language transcript tagging
• Multi-translation search with select-all, display-translation choice (drawer selection vs matched translation)
• Session context: bare "look at verse twelve" resolves against the passage currently live
• Dedicated AI Scripture settings tab, guided whisper install per platform, model downloads with progress/resume

Guarantees (please review these first)

• Off by default: no UI, stores, audio capture, downloads or network activity unless explicitly enabled; mic starts only on the operator's click
• Privacy: audio never leaves the device (transcription is fully local); only short transcript text windows go to the user's chosen LLM provider, only while listening with a key saved; keyless mode makes zero network calls; the transcript is memory-only
• Keys: stored in the ACCESS store (non-portable, never cloud-synced — same as the Canva credentials precedent); raw keys never reach the renderer (booleans only); flagged for a possible safeStorage follow-up
• Zero new npm dependencies (axios + yauzl already present; whisper.cpp is a runtime download/detection)
• Supply chain: Windows binary auto-download is SHA-256-pinned against the official whisper.cpp release; macOS/Linux use system installs (brew/package manager) or a user-picked binary — no unofficial mirrors

Platform matrix

Platform	Whisper
Windows x64	official release auto-download (checksum-verified)
macOS (Intel + Silicon)	brew install whisper-cpp detection or custom path
Linux	package manager detection or custom path

Testing

• 141 unit tests (vitest, colocated) over detection, transcription windowing, providers, whisper management, voice commands
• Field-tested live at a conference on macOS. Honest note: transcription accuracy with a laptop mic capturing room audio is limited — a direct feed from the sound desk (any input selectable in settings) is the intended setup and testing on that is ongoing. Detection precision is deliberately conservative (explicit references only auto-project; everything ambiguous becomes a suggestion).
• Would love testers on Windows/Linux and with mixer feeds.

Happy to split, adjust scope, or rework anything — I kept every touched upstream file minimal and matched existing patterns (LTC worklet, ffmpegManager, ACCESS store, Material inputs, i18n) throughout.

---

Updates since opening

• Local Ollama provider — fully offline quoted-verse detection (Gemma 3/4, Llama, Qwen, Mistral, Phi via ollama.com), same prompt/schema as the cloud providers, no API key; completes the everything-stays-local story
• Auto-projection fixes from live field testing: the manual-override pause now only triggers when the operator overrides a scripture the AI itself projected (ordinary song/slide changes no longer silently pause auto mode) and self-resumes after a minute
• Spoken-form coverage: "Matthew 12 4", "12, 4", "12. 4", "12-4", word numbers, and cued chapter-only ("turn to Matthew chapter five") all auto-project; a bare "Matthew 5" mid-sentence stays a suggestion
• Music gating: whisper wraps singing in ♪ and hallucinates lyrics — music segments are shown faded in the transcript but never feed detection
• Interpretation hardening: a declared "languages spoken" set constrains per-window language detection; out-of-set guesses are re-checked against the detection language
• Capture/UX: system-default mic auto-selection, speech-tuned capture (AGC + noise suppression), sample-rate-aware resampling, resumable checksum-verified downloads with live progress

Test suite is now at 141.

[josephomills]

Pushed a round of updates from live field testing at a conference, plus a fully-local AI option:

• Local (Ollama) provider — quoted-verse detection with Gemma 3/4, Llama, Qwen, Mistral or Phi running on the user's own machine: no API key, free, offline, and nothing ever leaves the device. Cloud providers remain optional alternatives.
• Auto mode fixes: the manual-override pause was too eager (any song/slide change paused auto-projection until manually resumed) — it now only triggers when the operator overrides a scripture the AI itself projected, and self-resumes after a minute.
• Real spoken forms auto-project: preachers rarely say the words "chapter" and "verse" — "Matthew 12 4" (and every punctuation whisper produces for it: "12, 4", "12. 4", "12-4", word numbers) plus cued chapter-only ("turn to Matthew chapter five") now project automatically; a lone "Matthew 5" mid-sentence still asks first.
• Music never triggers detection: whisper hallucinates lyrics for songs it hears (♪) — those segments show faded in the transcript but are excluded from detection and the LLM.
• Interpretation mode: declared languages constrain whisper's per-window guess; out-of-set windows are double-checked against the detection language.
• Plus capture-quality fixes (auto mic selection, speech-tuned gain/noise settings, sample-rate-aware resampling) and resumable, checksum-verified model downloads.

141 unit tests passing. The PR description's update section has the full list.

[AlloDel]

Left a longer note on #3547 about complementary fully local streaming work I have been testing (no API keys / no LLM for the core path). Happy to collaborate or wait for maintainer preference so we do not step on this review.

[josephomills]

@AlloDel your driver is merged and wired up — engine selector in settings, resumable model download, packaging config. Thank you, the VAD choice and the Electron notes were spot on.

One change to your driver, after live testing. The decoder deafness you documented goes deeper than stream reuse: any chunked feeding (fresh, reused, or persistent stream) intermittently decodes short utterances like "next verse" to nothing. The same samples in one batch acceptWaveform on a fresh stream decode correctly every time. So the driver now buffers each utterance (1s pre-roll) and batch-decodes when the VAD closes, waiting 0.5s of real tail audio first so a soft last word isn't cut mid-word. VAD retuned: threshold 0.3, min-silence 0.8s, min-speech 0.15s. If your live-testing notes contradict any of this, I'd like to know.

What I'd like you to review: downloading the sherpa runtime on demand — finishing your zero-setup goal. Right now the npm natives ship in the installer (~16 MB/platform) for everyone. The plan, mirroring our ffmpeg/whisper downloaders:

Exact-pin sherpa-onnx-node@1.13.4; exclude the platform packages from electron-builder, keep the JS wrapper bundled
Self-hosted per-platform ZIPs with pinned SHA-256s (plus proper license files — the npm tarballs omit them)
Extract to userData/bin/sherpa/ with sherpa-onnx-node/ copied beside it, so addon.js's own ../sherpa-onnx- probe resolves — no patching
macOS: clear quarantine + ad-hoc codesign after extraction (darwin-x64 ships its .node unsigned)
One "Enable" gate chains runtime (~7 MB) + model (662 MB) downloads
Questions:

Does the sibling-layout require() hold across sherpa-onnx-node versions you've used?
Anything from your live testing this misses?
Any concern with self-hosting the natives vs keeping them npm-managed?
Whisper stays the default; settings, multilingual and detection are unchanged.

[vassbo]

I will review this soon.

But just a couple things for now I would like changed:

• Can all the new locale strings be placed under their own parent key(s) at the bottom in en.json
• Can everything AI related have their own parent folders, so electron/ai/... & frontend/ai/...

[josephomills]

Alright @vassbo
Will do

[josephomills]

Since the last update — bubble & scripture polish

• The floating bubble now sits properly in the app's layering scheme, and the live transcript ticker wears a thin AI ring so it stays readable over any panel.
• The transcript is selectable, and can be copied from its context menu.
• Toggle semantics are locked down: turning AI off always stops a running listening
  session (the main switch used to leave it running invisibly), and it also turns
  AI Scripture off, since the feature needs the AI layer — while AI Scripture can
  still be turned off on its own. Loaded settings are sanitized the same way, so an
  older save can't wake up in the invalid "scripture on / AI off" state after a
  restart. The scripture options panel also follows changes made from anywhere now,
  instead of keeping a snapshot from when it mounted.

AI in Quick Search

The palette (Ctrl+G) has a new AI category — its own group header, ai: / ai
prefix filtering, and clicking the group header filters too:

• AI — opens the AI settings page
• Enable/Disable AI — the main switch, label follows the current state
• Manage models — opens the model manager on the AI settings tab
• Enable/Disable AI Scripture — the feature switch
• AI Scripture settings — opens the scripture drawer with its options panel
  already revealed (through a new generic drawer_options trigger any drawer tab
  can reuse)
• Start/Stop listening — only listed while the feature is enabled, following the
  live session state

The entries keep a fixed order, and reopening the palette re-runs the search so the
state-dependent labels never go stale. Terms like whisper, transcription,
voice, microphone, models, LLM or provider surface the right entries.

All 373 AI tests stay green, and the svelte-check/lint baselines are unchanged.

@vassbo I'll pause here and wait for you to have a look at it so we know the way forward.
@AlloDel you can also have a look when you have time so you can also give me feedback.

Thank you!

[vassbo]

@josephomills The plan is to review this soon, and get it merged as a hidden/experimental feature before the next beta release in a few weeks.

[josephomills]

Great! Looking forward to a lot of feedback from different speakers/preachers, different languages & different computers.

I'm also looking up ways to simplify this feature:
Eg a default multilingual speech model and improving STT quality. Already seeing massive improvements with the rework.