Matrix docs agent

An agentic RAG assistant for the Matrix protocol specification. It answers developer questions using the official Matrix spec and related protocol documents, giving source-backed explanations instead of relying on the model's built-in knowledge.

Built for developers working on Matrix clients, bots, application services, homeserver integrations, or other protocol-compatible software. See docs/motivation.md for the problem, users, and why an agentic RAG approach fits.

Architecture

A LangGraph agent (a FastAPI service) routes each request, decomposes it into subtasks, and fans out to a dedicated, modular RAG subgraph that retrieves and grades evidence from a ChromaDB index of the Matrix spec. A Streamlit UI visualises the agent's steps, and Ollama serves the open-source LLM and embedding models locally — no paid APIs. Full details in docs/architecture.md.

Prerequisites

• Docker + Docker Compose — the recommended way to run everything.
• For the local dev path instead: Python 3.13 (pinned in .python-version) and uv, plus a running Ollama.
• First run needs internet and downloads ~2 GB of model weights (into a persistent Docker volume) plus a prebuilt index (~hundreds of MB).

Quick start

Clone the repo, then pick one of the two options below.

Option A — Docker Compose (recommended)

cp .env.example .env
docker compose up -d ollama       # start the model server
./scripts/setup-models.sh         # pull generation + embedding models (once)
./scripts/download-index.sh       # fetch the prebuilt Chroma index into data/index/
docker compose up -d              # start agent + UI

Open the UI at http://localhost:8501. On later starts, just docker compose up -d (models and index persist).

Option B — Local (uv)

Run the models with the Ollama container (docker compose up -d ollama then ./scripts/setup-models.sh), or point OLLAMA_BASE_URL at a native Ollama. Then:

uv sync                           # install the workspace + dev/ingest/evals/notebook groups
cp .env.example .env
./scripts/download-index.sh       # fetch the prebuilt Chroma index into data/index/

# agent (FastAPI, port 8000):
uv run uvicorn matrix_docs_agent.main:app --host 0.0.0.0 --port 8000

# UI (Streamlit, port 8501) — in a second shell:
uv run streamlit run services/streamlit/src/matrix_docs_ui/app.py

Prefer to rebuild the index yourself instead of downloading it? See docs/data-pipeline.md.

Evaluation

Two decision-making components are scored against a 20-case golden set (data/questions.yaml) by a deterministic offline harness (uv run python -m evals). Highlights: the router hits 100% intent accuracy with the default granite3.3:2b (warm p50 5.5s), and retrieval reaches hit@k = 100% with recall ≈ 0.92 at the calibrated 0.3 threshold. Full methodology and per-model tables are in docs/evaluation.md.

Documentation

• docs/architecture.md — system and agent-graph architecture.
• docs/motivation.md — problem, objectives, users, and why agentic RAG.
• docs/data-pipeline.md — corpus construction, index, and model choices.
• docs/evaluation.md — router and retrieval evaluation results.
• docs/model-benchmark.md — generation-model speed benchmark.

License

This project is licensed under the GNU Affero General Public License v3.0 or later. See LICENSE.