🎙️ handy-mcp

An offline-first Model Context Protocol (MCP) server that brings professional-grade speech-to-text to your favorite AI tools (Claude, Gemini, Cursor, etc.).

Powered by Handy's local models, sherpa-onnx, and onnxruntime.

✨ Features

• 100% Offline: All transcription happens locally on your machine. No API keys, no data leaks, no latency.
• Handy Integration: Automatically discovers and uses models from your local Handy installation.
• Batch Processing: Transcribe entire folders of audio files with a single command.
• Universal Format Support: Transcribe Audio (MP3, WAV, M4A, OGG) AND Video (MP4, MKV, AVI, MOV). The server automatically extracts the audio track!
• Audio Enhancement: Optional volume normalization and silence removal for noisy recordings.
• High Performance: Uses optimized Whisper/Parakeet models via ONNX.

🚀 Quick Start

1. Prerequisites

• Node.js (v18+)
• ffmpeg (Optional, but required for non-WAV files)

2. Installation

git clone https://github.com/sudoax0n/handy-mcp.git
cd handy-mcp
npm install
npm run build

3. Configuration

🤖 Claude Desktop (Windows)

Add to %APPDATA%\Claude\claude_desktop_config.json:

{
  "mcpServers": {
    "handy": {
      "command": "node",
      "args": ["E:/handy-mcp/dist/index.js"]
    }
  }
}

💻 Claude Code (CLI)

Run the following command:

claude config add mcp handy node E:/handy-mcp/dist/index.js

♊ Gemini / Codex CLI

Add the server to your respective .mcp or tool configuration file:

{
  "handy": {
    "command": "node",
    "args": ["E:/handy-mcp/dist/index.js"]
  }
}

💻 Standalone CLI Usage

You can also use handy-mcp directly from your terminal without an MCP client. This is great for shell scripts or quick transcriptions.

List Models

node dist/index.js list

Transcribe a File

node dist/index.js transcribe "C:/path/to/audio.wav" --save md --enhance

Batch Process a Folder

node dist/index.js transcribe "C:/audio-folder" --save txt --concurrency 2

Option	Description
--model <id>	Specify a model ID to use (run list to see IDs).
--enhance	Apply volume normalization and silence removal.
--save <format>	Save results to a file (txt, md, json).
--concurrency <n>	(Batch only) Number of files to process in parallel.

🛠️ Tools

Tool	Description
list_models	List all locally available STT models from Handy.
transcribe_file	Transcribe a single audio file (WAV, MP3, etc.) to text. Supports save_results and save_format.
batch_transcribe	Transcribe an entire directory of audio files at once. Supports save_results and save_format.

🧠 Model Compatibility & Architecture

handy-mcp is designed to be highly compatible with local models, but behind the scenes, it uses a dual-engine architecture to support different model types:

1. Whisper Models (Standard Path)

• Engine: sherpa-onnx-node
• Supported Models: Whisper Large-v3 (INT8), Whisper Medium, etc.
• How it works: Whisper models are fully compatible with the standard sherpa-onnx C++ engine. This is the recommended path for maximum stability and speed. It handles decoding natively using highly optimized C++ binaries.

2. Parakeet TDT Models (Direct Inference Path)

• Engine: onnxruntime-node (Custom Pipeline)
• Supported Models: Parakeet TDT 0.6B v3 (INT8), etc.
• How it works: The War Story. During development, we discovered that sherpa-onnx cannot run TDT (Token-and-Duration Transducer) architectures because its C++ engine hardcodes an autoregressive decoder loop. Since TDT models are non-autoregressive, they crash the engine with an Exit Code 1.
• The Fix: We built a custom "Direct Inference" pipeline that completely bypasses sherpa-onnx. When you use a Parakeet TDT model, handy-mcp loads the three raw ONNX files (nemo128.onnx feature extractor -> encoder-model -> decoder_joint-model) directly into Node.js using onnxruntime-node and executes a custom CTC-style greedy decoding loop in JavaScript.

If you experience any issues with Parakeet models, simply download a Whisper model via the Handy app, and handy-mcp will automatically switch to the stable sherpa-onnx engine!

⚙️ Configuration Options (Tools)

When calling the transcription tools, you can use these optional parameters:

• save_results: (boolean) Set to true to save the output to a file next to the original audio.
• save_format: (string) Choose between txt, md (markdown), or json. Default is txt.
• skip_existing: (boolean) Batch only. Skips transcription if a saved file already exists. Huge time saver! Default is true.
• enhance_audio: (boolean) Uses ffmpeg to normalize volume and remove silence before transcribing. Improves accuracy for noisy/quiet files. Default is false.
• concurrency: (number) Batch only. Number of files to process in parallel (1-5). Default is 1. Sequential processing is recommended for best stability on most local systems.

💬 Prompting Examples

Once connected, you can give your AI powerful audio-related instructions:

Basic Transcription

• "Transcribe the meeting recording at C:\Users\G\Desktop\meeting.mp3"
• "What did I say in my last voice note? E:\audio\note_01.wav"

Analysis & Action

• "Summarize the transcription of E:\audio\lecture.m4a into 5 bullet points."
• "Read E:\audio\interview.wav, transcribe it, and find any mention of project deadlines."
• "Transcribe E:\audio\feedback.mp3 and draft a response email based on the content."

Batch Power-Moves

• "Transcribe all the .wav files in E:\Handy\recordings, save the results as .md files, and give me a summary of everything mentioned."
• "Look through E:\Project\Audio, transcribe everything, and tell me if there are any conflicting instructions in the recordings."

🌟 GitHub Coolness

This project is the ultimate bridge between the Handy speech-to-text engine and the MCP ecosystem. It transforms your local AI into a professional transcriptionist that respects your privacy. No cloud, no costs, just pure local intelligence.

---

Built with ❤️ for the open-source community.