Skip to content

plugins.md

Latest commit

 

History

History
199 lines (145 loc) · 6.81 KB

plugins.md

File metadata and controls

199 lines (145 loc) · 6.81 KB
Raw

Plugin Development

Plugins are Python packages that extend Modelship with custom inference backends. Each plugin is a self-contained uv workspace package with its own dependencies, installed on demand.

Plugins can implement any usecase — TTS, STT, chat, embeddings, translation, image generation — not just speech synthesis.

How plugins work

When loader: custom is set in models.yaml, Modelship imports the module named by plugin and expects it to expose a ModelPlugin class extending BasePlugin.

A plugin overrides only the create_* method(s) matching its usecase:

Usecase Method to override Raw return type
tts create_speech RawSpeechResponse or AsyncGenerator[(bytes, int), None]
transcription create_transcription RawTranscription
translation create_translation RawTranslation
generate create_chat_completion RawChatCompletion or AsyncGenerator[RawChatDelta, None]
embed create_embedding list[list[float]]
image create_image_generation list[bytes] (PNG-encoded)

Plugins return protocol-agnostic raw outputs. The serving wrappers in modelship/infer/custom/openai/ translate these into OpenAI-compatible responses, so a different protocol adapter (e.g. Anthropic, gRPC) could be added later without touching any plugin.

Unimplemented methods fall back to a 404 "plugin does not support this action" error.

Creating a plugin

1. Create the package structure

plugins/
  myplugin/
    pyproject.toml
    myplugin/
      __init__.py
      myplugin.py

2. Write pyproject.toml

[project]
name = "myplugin"
version = "0.1.0"
requires-python = "==3.12.10"
dependencies = [
    # only packages unique to your plugin — see "Dependency contract" below
]

[build-system]
requires = ["uv_build"]
build-backend = "uv_build"

[tool.uv.sources]
modelship = { workspace = true }

[tool.uv.build-backend]
module-name = "myplugin"
module-root = ""

Dependency contract

Plugins assume the host environment provides modelship itself plus the full core/gpu/cpu stack: torch, torchvision, transformers, numpy, scipy, librosa, soundfile, onnxruntime[-gpu], diffusers, vllm. Do not redeclare any of these in your plugin's dependencies.

Why: plugin wheels are shipped to Ray workers via runtime_env, which installs them into a per-job venv layered over a base image that already has the core stack baked in. Redeclaring modelship or torch causes pip to either pull a second copy from PyPI (version drift, two packages on sys.path) or error on the layered install.

Declare only what's unique to your plugin — e.g. kokoro-onnx, pywhispercpp, snac. Dev-time imports of modelship.* resolve because the workspace's shared .venv always has the root modelship package installed; you don't need to declare it to get IDE/pyright support.

3. Implement ModelPlugin

TTS example

# plugins/myplugin/myplugin/myplugin.py
from collections.abc import AsyncGenerator

from modelship.plugins.base_plugin import BasePlugin
from modelship.infer.infer_config import ModelshipModelConfig
from modelship.openai.protocol import ErrorResponse, RawSpeechResponse


class ModelPlugin(BasePlugin):
    def __init__(self, model_config: ModelshipModelConfig):
        self.model_name = model_config.model
        self.config = model_config.plugin_config or {}

    async def start(self):
        # load your model here
        pass

    async def create_speech(
        self,
        input: str,
        voice: str | None = None,
        speed: float | None = None,
        stream: bool = False,
        request_id: str | None = None,
    ) -> RawSpeechResponse | AsyncGenerator[tuple[bytes, int], None] | ErrorResponse:
        audio_bytes = b"..."  # your synthesis here
        return RawSpeechResponse(audio=audio_bytes)

STT example

from modelship.plugins.base_plugin import BasePlugin
from modelship.infer.infer_config import ModelshipModelConfig
from modelship.openai.protocol import ErrorResponse, RawTranscription


class ModelPlugin(BasePlugin):
    def __init__(self, model_config: ModelshipModelConfig):
        self.model_name = model_config.model

    async def start(self):
        pass

    async def create_transcription(
        self,
        audio_data: bytes,
        language: str | None = None,
        prompt: str | None = None,
        temperature: float | None = None,
        request_id: str | None = None,
    ) -> RawTranscription | ErrorResponse:
        text = "..."  # your transcription here
        return RawTranscription(text=text, language=language, duration_seconds=0.0)

4. Export ModelPlugin from __init__.py

# plugins/myplugin/myplugin/__init__.py
from myplugin.myplugin import ModelPlugin

__all__ = ["ModelPlugin"]

5. Register the extra in the root pyproject.toml

[project.optional-dependencies]
myplugin = ["myplugin"]

[tool.uv.sources]
myplugin = { workspace = true }

6. Install and configure

uv sync --extra myplugin

In models.yaml:

- name: myplugin
  usecase: tts        # or transcription, translation, generate, embed, image
  loader: custom
  plugin: myplugin
  num_gpus: 0.1

SSE streaming (TTS)

For streaming speech, yield (pcm_bytes, sample_rate) tuples from an async generator. pcm_bytes must be signed 16-bit little-endian mono PCM — the serving wrapper base64-encodes each chunk into SSE speech.audio.delta events.

async def create_speech(self, input, voice=None, speed=None, stream=False, request_id=None):
    if stream:
        return self._stream(input, voice, speed)
    # non-stream path
    audio = self._synthesize_full(input)
    return RawSpeechResponse(audio=audio)

async def _stream(self, input, voice, speed):
    for pcm_chunk, sample_rate in self._synthesize_chunks(input):
        yield pcm_chunk, sample_rate

Plugin README

Every plugin must include a README.md in its package root (plugins/myplugin/README.md). This is the primary documentation for users configuring the plugin. It should cover:

  • Installation — how to install the plugin (uv sync --extra for local development; automatic via wheels for deployment)
  • Configuration — example models.yaml entry with all plugin_config options documented in a table
  • Voices / options — any model-specific choices (voice presets, providers, etc.)
  • Example request — a working curl command

See the built-in plugins for reference: Kokoro ONNX, Orpheus, whisper.cpp.

Submitting to this repo

Open a PR adding:

  • plugins/myplugin/ with your package
  • plugins/myplugin/README.md documenting configuration and usage
  • One line in root pyproject.toml optional extras