Live Whisper

Real-time streaming speech-to-text server using faster-whisper (CTranslate2) on GPU, served over a WebSocket, with a browser page that streams your microphone to it.

For a non-browser backend that wants to stream audio to this service programmatically (protocol details, auth, and how to pick between multiple instances by load), see CONNECTING.md.

How it works

• server.py — the transcription API. FastAPI app exposing /ws, a WebSocket per connection. Each connection streams raw 16kHz mono PCM16 audio; the server buffers it, periodically re-transcribes the trailing window for live "partial" captions, and commits a "final" line once it detects ~1.2s of silence. Runs on port 8000.
• webserver.py — just serves the static webpage (static/index.html) on its own port, separate from the API. Runs on port 8080.
• static/index.html — captures mic audio in the browser, downsamples to 16kHz PCM16, and streams it over a WebSocket to the API server's port (hardcoded in the page as API_PORT, currently 8000 — change it there if you run the API on a different port). Includes a language slider (Italian/English/Spanish/French/German) that's sent as ?lang= on the WebSocket URL — the server transcribes in that fixed language for the whole session (no auto-detection; it was unreliable on short rolling windows).

Model

Using Na0s/Medical-Whisper-Large-v3 — a large-v3 fine-tune on doctor/patient consultations — converted to CTranslate2 format locally at models/medical-whisper-large-v3-ct2/ (that's what WHISPER_MODEL points to by default). ./setup.sh --with-model does the conversion below for you; to run the steps by hand instead:

uv run python -c "
from huggingface_hub import snapshot_download
snapshot_download('Na0s/Medical-Whisper-Large-v3', local_dir='models/medical-whisper-large-v3-hf')
"
# that repo ships slow-tokenizer files only; faster-whisper needs tokenizer.json
uv run python -c "
from transformers import WhisperTokenizerFast
tok = WhisperTokenizerFast.from_pretrained('models/medical-whisper-large-v3-hf')
tok.save_pretrained('models/medical-whisper-large-v3-hf')
"
uv run ct2-transformers-converter \
  --model models/medical-whisper-large-v3-hf \
  --output_dir models/medical-whisper-large-v3-ct2 \
  --copy_files tokenizer.json preprocessor_config.json \
  --quantization float16 --force

The conversion step needs transformers/torch/ctranslate2, which live in the convert optional dependency group (not installed by a plain uv sync) — use uv sync --extra convert if you're running the commands above by hand.

Community medical fine-tune — verify accuracy against your own recordings before relying on it; treat output as requiring clinician review, not autonomous documentation. To fall back to stock Whisper, set WHISPER_MODEL=large-v3 (or small, etc.) — faster-whisper will download it from Hugging Face automatically.

Setup

Current GPUs support efficient float16 execution in CTranslate2, so compute_type=float16 is the default everywhere (model conversion, WHISPER_COMPUTE) — faster and lower VRAM than float32 with negligible accuracy loss. (Earlier hardware in this deployment was Pascal, which doesn't support efficient float16/int8_float16 — if you ever move back to Pascal-class GPUs, use WHISPER_COMPUTE=float32 instead and reconvert the model with --quantization float32.)

This project uses uv for Python dependency management — one lockfile (uv.lock), no manual venv juggling. The unified setup script handles everything (installing uv if it's missing, syncing dependencies, generating .env and the TLS cert, and optionally the model):

./setup.sh                # deps + .env + TLS cert, no model download
./setup.sh --with-model    # also download + convert the medical model
./setup.sh --ip 1.2.3.4    # pin the IP baked into the cert's SAN

It's idempotent — re-run it any time; it only fills in what's missing and leaves .env and existing certs untouched. Equivalent manual steps, if you'd rather do it yourself:

uv sync   # creates .venv and installs deps from uv.lock
cp .env.example .env   # then fill in WHISPER_API_TOKEN — see "Authentication" below

The medical model above needs converting once (see "Model" section). Stock Whisper model weights download automatically from Hugging Face on first run (needs internet the first time; cached under ~/.cache/huggingface after that).

TLS (required for the mic to work over the network)

Browsers only expose getUserMedia (mic access) on https:// or localhost origins. Since the page is reached over the network at a non-localhost address, it must be served over HTTPS or the mic prompt silently never appears. A self-signed cert is already generated at certs/cert.pem / certs/key.pem (covers this machine's LAN/public IP, 127.0.0.1, and localhost).

./setup.sh checks this on every run: it compares the cert's IP SAN against the box's current IP and regenerates the cert automatically if they've drifted apart (e.g. after a DHCP lease change) — no separate step needed. It writes to a temp file and renames over the old one, so it works even if certs/ was previously set up with root-owned files. Pass --ip 1.2.3.4 to pin a specific IP instead of auto-detecting.

To regenerate it by hand instead:

MY_IP=$(ip -4 addr show enp179s0f0 | grep -oP 'inet \K[\d.]+')
openssl req -x509 -newkey rsa:2048 -nodes -keyout certs/key.pem -out certs/cert.pem -days 365 \
  -subj "/CN=whisper-live" \
  -addext "subjectAltName=IP:$MY_IP,IP:127.0.0.1,DNS:localhost"

Running

Two separate processes: the transcription API, and the webpage. Both need --ssl-keyfile/--ssl-certfile since both are reached from outside localhost.

.env has API_PORT/WEB_PORT/STATUS_PORT entries (see "Config via environment variables" below), but outside Docker Compose only STATUS_PORT (as WHISPER_STATUS_PORT) is read automatically — server.py opens that port itself. API_PORT/WEB_PORT only drive docker-compose.yml's port mapping; for a bare run, pass --port explicitly as shown below (matching .env's values if you want them consistent between the two run modes).

# terminal 1 — transcription API (port 8000). Defaults already use all 3 GPUs.
uv run uvicorn server:app --host 0.0.0.0 --port 8000 \
  --ssl-keyfile certs/key.pem --ssl-certfile certs/cert.pem

# terminal 2 — webpage (port 8080)
uv run uvicorn webserver:app --host 0.0.0.0 --port 8080 \
  --ssl-keyfile certs/key.pem --ssl-certfile certs/cert.pem

To stop them: pkill -f "uvicorn server:app" and pkill -f "uvicorn webserver:app" (or kill <pid> from ps aux | grep uvicorn).

Then open https://<this-machine-ip>:8080/ (note https) from any browser on the network, click Start, allow mic access, and talk. The page connects out to port 8000 (also https/wss) for the actual transcription WebSocket. Your browser will warn about the self-signed certificate on both origins (8080 for the page, 8000 for the API) — click through "Advanced → Proceed" once for each; this is expected since it's not a CA-signed cert.

--host 0.0.0.0 is what makes each one reachable from other machines, not just localhost. If this box has a firewall (ufw, cloud security group, etc.), open ports 8000, 8080, and 8001 (see "Concurrency notes" for the third one) for whichever clients need access.

If you serve the webpage on a different port than 8000, update API_PORT at the top of the <script> in static/index.html to match wherever server.py is actually running.

Running with Docker

Dockerfile + docker-compose.yml package both processes as containers built from the same image (nvidia/cuda:12.4.1-runtime-ubuntu22.04 + uv + pyproject.toml/uv.lock). The image installs dependencies with uv sync --locked, so the container gets the exact versions pinned in uv.lock. The models/ and certs/ directories are bind-mounted read-only rather than baked into the image — the model weights are multiple GB and the certs contain a private key, neither belongs in an image layer.

One-time host setup: NVIDIA Container Toolkit

The transcription container needs GPU access, which requires Docker's nvidia runtime. This is a one-time install on the host (not inside any container):

# 1. Add NVIDIA's apt repo and signing key
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# 2. Install the toolkit
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

# 3. Register the `nvidia` runtime with Docker and restart the daemon
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# 4. Verify — should print all GPUs (e.g. the 3x GTX 1080 Ti on this box)
sudo docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

If you're on a very new Ubuntu release and apt-get update complains the repo's codename is unrecognized, NVIDIA's list is a generic (non-codename-pinned) deb repo, so this is usually transient — retry, or check https://github.com/NVIDIA/nvidia-container-toolkit/issues for current guidance.

You'll also want your user in the docker group so you don't need sudo for every docker command: sudo usermod -aG docker $USER, then log out and back in.

Build and run

The API (api) and webpage (web) are separate Compose services behind profiles, so they can be started independently — a bare docker compose up with no profile starts nothing:

# transcription API only (port 8000, GPU)
sudo docker compose --profile api up -d

# webpage only (port 8080)
sudo docker compose --profile web up -d

# both
sudo docker compose --profile api --profile web up -d

docker compose builds the image automatically on first run; rebuild after changing server.py, webserver.py, static/, pyproject.toml, or uv.lock with:

sudo docker compose build

Logs: sudo docker compose logs -f api (or web). Stop: sudo docker compose --profile api --profile web down.

Both containers read certs/key.pem / certs/cert.pem from the host via bind mount, so generate the TLS cert (see "TLS" above) before starting them. The api container reads models/ the same way, so the model must already be converted/downloaded on the host first (see "Model" above) — nothing is downloaded inside the container.

WHISPER_MODEL, WHISPER_DEVICE, WHISPER_COMPUTE, WHISPER_DEVICE_INDEX, WHISPER_NUM_WORKERS, and STATUS_PORT (see table below) are all overridable without editing the compose file — either export them in your shell before docker compose up, or put them in a .env file next to docker-compose.yml:

# .env
WHISPER_NUM_WORKERS=2
WHISPER_API_TOKEN=<your-secret>

WHISPER_API_TOKEN isn't optional under Compose — the api service is deliberately set to fail fast (${WHISPER_API_TOKEN:?...}) rather than start unauthenticated. See "Authentication" below for what this protects and how the webpage uses it.

Running behind nginx-proxy (no self-signed cert)

The api/web profiles above use a self-signed cert (see "TLS"), so browsers show an untrusted-certificate warning that has to be clicked through — expected, but annoying, and it means the token travels to first-time visitors with a scarier prompt than it should. If this box already runs the shared nginx-proxy + acme-companion stack (as Yggdrasil-prod and other projects here do — check docker network ls for a proxy-net and docker ps for nginx-proxy), a proxy profile does the same thing: no cert of this project's own, real Let's Encrypt TLS terminated by the shared proxy.

PROXY_DOMAIN must be an unclaimed subdomain pointing at this host — not pdor.ing.unimore.it. That one is already served by a different app on the shared proxy (confirmed via its Strict-Transport-Security response header, which also happens to be why the self-signed cert on WEB_PORT/API_PORT gets a hard, unbypassable warning in Chrome if you've ever loaded that hostname over HTTPS: HSTS applies per-hostname, not per-port, once set). Get a fresh subdomain (e.g. whisper.ing.unimore.it) pointed at this host before using this profile.

# .env
PROXY_DOMAIN=whisper.ing.unimore.it   # an UNCLAIMED subdomain, already resolving to this host
LETSENCRYPT_EMAIL=you@example.it      # required for acme-companion to request a cert

sudo docker compose --profile proxy up -d

This starts api-proxy and web-proxy instead of api/web — same images, no published ports, no cert volume mount. Both join the external proxy-net network and set VIRTUAL_HOST/LETSENCRYPT_HOST to PROXY_DOMAIN; nginx-proxy picks them up via docker-gen and acme-companion issues/renews the cert automatically.

One hostname, two services — routed by path, not port. nginx-proxy routes by hostname (and optionally path), not by port, so both services share PROXY_DOMAIN. api-proxy sets VIRTUAL_PATH=/ws,/load so those two paths route to it; everything else (i.e. /, the static page) falls through to web-proxy's catch-all. This requires changing static/index.html: API_PORT at the top of its <script> (currently 9097, used to build `${proto}://${location.hostname}:${API_PORT}/ws?...`) needs to become just ${location.hostname}/ws?... with no port, since there's no separate API port under this hostname anymore — port 443 (implicit) reaches both services, and the path is what distinguishes them. That edit hasn't been made in this checkout; do it before relying on this mode.

Both PROXY_DOMAIN and LETSENCRYPT_EMAIL default to empty if unset, which leaves VIRTUAL_HOST/LETSENCRYPT_HOST empty too — nginx-proxy simply won't create a vhost for a container in that state (no hard failure, just silently not proxied), so double-check docker compose --profile proxy config shows the hostname you expect before assuming it's live.

Config via environment variables

Var	Default	Meaning
WHISPER_MODEL	models/medical-whisper-large-v3-ct2	Local CTranslate2 model path, or a stock size (tiny/small/medium/large-v3) to download from Hugging Face instead.
WHISPER_DEVICE	cuda	cuda or cpu.
WHISPER_COMPUTE	float16	CTranslate2 compute type. Current GPUs support float16 efficiently — see "Setup". Use float32 only on older (Pascal-class) GPUs that don't.
WHISPER_DEVICE_INDEX	0,1,2	Comma-separated GPU indices to replicate the model across — defaults to all 3 GPUs on this box. CTranslate2 does not spread across GPUs unless told to.
WHISPER_NUM_WORKERS	4	Concurrent CTranslate2 inference workers per GPU in WHISPER_DEVICE_INDEX. Total concurrent capacity ≈ WHISPER_NUM_WORKERS × (GPUs in WHISPER_DEVICE_INDEX). Re-benchmark VRAM headroom per worker on current hardware before pushing this up.
WHISPER_API_TOKEN	(required, no default)	Shared-secret token required to open /ws, set in .env — see "Authentication" below.
API_PORT	8000	Compose-only: host+container port for the api service (/ws and /load). Under a bare uv run uvicorn (no Compose), this isn't read automatically — pass --port yourself, matching whatever you use here.
WEB_PORT	8080	Compose-only: host+container port for the web service (the static page). Same caveat as API_PORT for bare runs — pass --port yourself.
WHISPER_STATUS_PORT (bare run) / STATUS_PORT (Compose)	8001	Port GET /load listens on — separate from API_PORT/WHISPER_API_TOKEN's /ws port, so status can be polled independently (e.g. through a firewall/LB rule that doesn't expose /ws). Unlike API_PORT/WEB_PORT, server.py reads this one itself (via WHISPER_STATUS_PORT) even outside Compose, since it's opened by the app, not passed to uvicorn on the CLI. Set to 0 to disable the second listener entirely. Uses TLS automatically if certs/cert.pem/certs/key.pem exist (paths overridable via WHISPER_CERT_FILE/WHISPER_KEY_FILE), otherwise plain HTTP.
PROXY_DOMAIN	(empty)	proxy profile only (see "Running behind nginx-proxy"). Hostname nginx-proxy/acme-companion issue the Let's Encrypt cert for and route to api-proxy/web-proxy. Must already resolve to this host. Left empty, the services just don't get a vhost — no crash, just not proxied.
LETSENCRYPT_EMAIL	(empty)	proxy profile only. Passed to acme-companion for the ACME account backing the cert. Optional in principle (Let's Encrypt doesn't require an email) but recommended so you get expiry-related notices.

Concurrency notes

• GET /load on WHISPER_STATUS_PORT (default 8001, independent of the main /ws port) reports current utilization as JSON: active_connections (open WebSocket sessions), active_transcriptions (transcribe() calls in flight right now), transcribe_capacity (WHISPER_NUM_WORKERS × GPUs in WHISPER_DEVICE_INDEX), and load_ratio (active_transcriptions / transcribe_capacity). No auth required — it reveals no session content, only aggregate counts. Useful for a client, or a load balancer sitting in front of several instances of this server, to check how saturated one instance is before sending it more work. It's a second listener in the same process (started via uvicorn.Server(...).serve() as an asyncio task on startup), not a separate replica — it shares this instance's counters, so it goes down if this instance does.
• Each WebSocket connection keeps its own audio buffer — multiple people can connect and transcribe independently and simultaneously.
• Transcription calls run in a thread pool (WHISPER_NUM_WORKERS threads) so the asyncio event loop stays free to keep receiving audio from every connection while GPU inference is in progress.
• num_workers on WhisperModel tells CTranslate2 to actually run that many inference calls concurrently against the one loaded set of weights (rather than serializing them), so raising it is what lets the GPU do several transcriptions "insieme". Re-check VRAM headroom for the current float16 config before pushing WHISPER_NUM_WORKERS up — don't push further without checking for OOM.
• All 3 GPUs are used by default (WHISPER_DEVICE_INDEX=0,1,2) — confirmed via nvidia-smi that this actually allocates memory on each listed device, not just GPU 0. Combined with WHISPER_NUM_WORKERS, total concurrent transcription capacity is num_workers × number of GPUs listed (12 by default).
• Measured throughput on the earlier small/int8 config: saturates around ~11 calls/sec, supporting roughly 16 concurrent active speakers before live captions start lagging. The medical large-v3 model is slower per call, so that ceiling is lower now — re-benchmark if you need a precise number for this model.

Authentication

/ws requires a shared-secret token — each connection burns real GPU time, so it must not be left open to anyone who can reach the port. The token is passed as ?token= on the WebSocket URL (browsers' native WebSocket API can't set custom headers, so a query param is the practical option here) and checked with a constant-time comparison (secrets.compare_digest) before the connection is accepted; an invalid or missing token gets the handshake refused with close code 1008.

The token lives in a .env file in the project root (gitignored — never commit it), loaded automatically by both server.py (via python-dotenv) and docker compose (which reads .env natively). server.py refuses to start if WHISPER_API_TOKEN is missing entirely, so it can never come up silently unauthenticated.

cp .env.example .env
python3 -c "import secrets; print(secrets.token_urlsafe(32))"   # paste into .env

A real .env already exists in this checkout with a generated token — rotate it any time by generating a new value and pasting it in; both server.py and any running Compose containers need a restart to pick up the change.

The webpage (static/index.html) has an "API token" field, prefilled automatically: webserver.py reads WHISPER_API_TOKEN from its own .env (same convention as server.py) and substitutes it into the page at request time, so nothing needs pasting in for the common case of one page talking to one API instance using the same .env. The field stays editable and whatever's typed there is kept in that browser's localStorage and takes priority over the server-supplied value on future visits — useful if you're pointing this page at a different API instance with a different token. Under Docker Compose, webserver.py's container needs WHISPER_API_TOKEN passed to it too (already wired up in docker-compose.yml's web/web-proxy services) — without it the field prefills empty, same as if the var were unset entirely.

This only guards the WebSocket endpoint. The static webpage itself (webserver.py, port 8080) still has no auth of its own — anyone who can load the page now gets the token without even needing to paste it, so reachability of the page is the actual access boundary now, not the manual-paste step that used to exist. Keep it behind HTTPS (see "TLS" above) so the token isn't sent in the clear, and for anything beyond a trusted LAN, put both ports behind a reverse proxy (nginx/Caddy, or the proxy profile above) with its own access control as defense in depth.