Skip to content

AGENTS.md

Latest commit

 

History

History
92 lines (65 loc) · 12.4 KB

AGENTS.md

File metadata and controls

92 lines (65 loc) · 12.4 KB
Raw

AGENTS.md

Canonical agent-instruction file for this repository. Both Claude Code (via the @AGENTS.md import in CLAUDE.md) and OpenCode (which reads AGENTS.md natively) load this file. It carries the always-on rules; deep reference documentation lives under docs/ and is read on demand, not loaded into context (see Reference at the end).

What this is

A PowerShell wrapper around upstream llama.cpp, pinned as a submodule at vendor/llama.cpp/. No original C/C++ lives here — only .ps1 scripts driving CMake + MSVC + Conda, plus one benchmark helper in examples/mtp-bench.py. The session shell is bash on Windows; the project's own scripts must run from pwsh/powershell.

Commands

./rebuild_llama.cpp.ps1                          # auto-detects CUDA vs OpenBLAS
./rebuild_llama.cpp.ps1 -version "b1138"         # pin a tag / commit
./rebuild_llama.cpp.ps1 -pullRequest "18675"     # build a PR
./rebuild_llama.cpp.ps1 -target "llama-server"   # CMake target subset
./rebuild_llama.cpp.ps1 -blasAccelerator OFF     # OpenBLAS | CUDA | OFF

./examples/server.ps1 -model ".\vendor\llama.cpp\models\<x>.gguf"
Get-Help -Detailed ./examples/server.ps1         # full option list

. .\load_env.ps1; llama-server                   # router mode, configured by ./.env

Binaries land in ./vendor/llama.cpp/build/bin/Release/. Conda env llama.cpp (Python 3.12) must already exist — the scripts call conda activate llama.cpp themselves.

Router mode reads its launch configuration from .env in the project root (gitignored; template .env.example). load_env.ps1 parses it the same way windows_manage_large_language_models does — split on the first =, skip blank and # lines, no quoting, no trimming — and sets each key as an environment variable. Keep it to CUDA_* and server-scoped LLAMA_ARG_* keys; unscoped ones like LLAMA_ARG_CTX_SIZE would reach every llama.cpp binary.

Other helpers in examples/: count_tokens.ps1, benchmark.ps1 (perplexity), speculative_decoding.ps1, speed-bench.ps1 (router-mode throughput sweep), mtp-bench.py. README.md -> Usage is the end-user inventory of what they do; don't restate their flags here. Get-Help works on server.ps1, speed-bench.ps1 and count_tokens.ps1 only — the other two carry no comment-based help.

No tests, no linter. Verify changes by running an example script against a real GGUF model.

Non-obvious behavior

  • The submodule always shows dirty. rebuild_llama.cpp.ps1 prepends an idempotent OpenBLAS linking shim to vendor/llama.cpp/CMakeLists.txt; .gitmodules sets ignore = dirty for it. Don't "clean it up." docs/build_system.md -> Submodule lifecycle
  • Each build wipes vendor/llama.cpp back to origin/master then checks out the requested -version / PR, so local edits there are lost by design. Other submodules are never advanced by the build script and must be bumped by hand. docs/build_system.md -> Submodule lifecycle
  • ./patches/*.patch is the only supported way to carry a local change into the submodule. It is re-applied after every checkout and the build aborts if a patch no longer applies. Currently one patch, which keeps GGUF conversion from stalling when a tensor's row width puts the quantizer group on a slow allocation size. docs/build_system.md -> Local patches
  • CUDA is selected iff both nvidia-smi and nvcc are on PATH. Missing either silently falls back to OpenBLAS.
  • Three vendored paths are hardcoded (gguf_dump.py, speed-bench/, models/templates/). Upstream has moved them before; after a version bump treat a startup failure naming one as a relocation first. docs/build_system.md -> Upstream path dependencies
  • server.ps1 -additionalArguments splits on whitespace and re-pairs tokens into key/value flags. Values that contain spaces will not survive this parser.
  • Rebuild aborts on running build-tree processes. Before any destructive op, rebuild_llama.cpp.ps1 checks Get-Process for any EXE under vendor/llama.cpp/build/ and throws with the PID list. Catches the forgot-to-stop-llama-server.exe case.
  • Keep ~400 MiB free on the display GPU. Below ~300 MiB WDDM pages the working set to system RAM and the entry silently runs 20-45 % slower; nothing in the llama.cpp log shows it, only throughput does. Check nvidia-smi free memory after a preset change. docs/presets.md -> Device pinning and multi-GPU

Traps

Prohibitions that cause a silent OOM, silent corruption, or a startup abort. Each is stated here without rationale so it is always in context; read the linked section before acting on one. These lines are a deliberate projection of docs/ — when a trap changes, both move together.

  • Never pair direct-io with no-mmap; use load-mode = dio — except on Qwen3.8-Flash-Next, which must keep load-mode = mmap. docs/presets.md -> load-mode
  • Never set mmproj-offload = true on a tier where LLM + KV already saturate VRAM. docs/presets.md -> mmproj-offload
  • Never set swa-full on a DeepSeek-V4-Flash or Muse Glimmer entry. docs/presets.md -> swa-full
  • Never set context-shift on a Muse Glimmer entry; it is silent corruption, not a refusal. docs/model_tuning/muse-glimmer.md
  • Never add RoPE scaling to a Muse Glimmer entry. docs/model_tuning/muse-glimmer.md
  • no-host = true is mandatory on the DeepSeek and Qwen3.8-Flash-Next entries, and on any entry that pushes tens of GiB of experts to CPU; without it the load fails as a misleading CUDA OOM. docs/presets.md -> no-host
  • Keep fit = on on the DeepSeek and Qwen3.8-Flash-Next entries; never add n-cpu-moe/-ot, and never set n-gpu-layers to anything but -1 — fit then silently no-ops. docs/presets.md -> fit
  • cache-type-k and cache-type-v must be identical on any MLA entry — currently deepseek4 and Ling-3.0-tiny; differing values are startup-fatal. docs/model_tuning/deepseek-v4-flash.md, docs/model_tuning/ling-3.0.md
  • Never set image-min-tokens on a gemma-4 entry; it is a qwen3vl_merger key only. docs/model_tuning/gemma-4.md
  • Never drop a chat-template-file pin; it replaces the GGUF-embedded template and is not redundant with jinja = true. docs/model_tuning/qwen.md, docs/model_tuning/gemma-4.md
  • Quantize Qwen3.8 GGUFs from Qwen/Qwen3.8-27B, never from the derived -FP8 repo. docs/model_tuning/qwen.md
  • Any cache-type-k/cache-type-v/-draft pair added to a preset must also be added to -DGGML_CUDA_FA_QUANTS in rebuild_llama.cpp.ps1 and the build re-run; the flag no longer compiles all combinations, and neither the server log nor test-backend-ops reports a missing pair. docs/build_system.md -> CUDA build flags
  • Never build a revision below b10876 without swapping -DGGML_CUDA_FA_QUANTS for -DGGML_CUDA_FA_ALL_QUANTS=ON by hand; the flag does not exist there, CMake takes it as an unused cache entry, and the six q5_0-q4_1 entries lose flash attention to the CPU backend. Only -version / -pullRequest can reach this. docs/build_system.md -> CUDA build flags

Version Control

  • Work lands on develop; main receives it only through a pull request. Every commit that changes behavior carries its own CHANGELOG.md version entry in the same commit. docs/conventions.md -> Commit messages
  • A commit subject is one imperative line with no prefix and no ticket key; the body wraps at 80 columns and carries the measurement, the rationale, the alternatives that were dropped and what was deliberately left undone. No Co-Authored-By, Generated with or Signed-off-by trailer. docs/conventions.md -> Commit messages
  • A pull request body answers what changed, why, the shortcomings of the approach, which feedback you want and what is not done. A link supplements it and never carries it. docs/conventions.md -> Pull request descriptions
  • Commit and push are never automatic: commit only when the user asks, push only when the user asks, and "commit" does not imply "push".

Documentation and prose

  • Every rule has one home: AGENTS.md carries the invariant, the matching docs/ file the contract, the commit message the decision, and every other mention is a pointer in the form `docs/<file>.md` -> *Section*. Never write down a version, an inventory or a value nobody reasons about; point at the file that sets it. docs/conventions.md -> Documentation
  • Size budgets, in bytes: AGENTS.md about 12,000 and never over 16,000; a reference document under docs/ about 30,000 and never over 45,000. docs/conventions.md -> Size budgets
  • Pad every cell of a markdown table, and keep the em dash for a genuine break in thought rather than as a default joiner. A comment says why, never what, and carries no history. docs/conventions.md -> Punctuation and tables, Comments

Changelog

  • A bullet records a change a reader can observe by running the repository, one per change and never one per commit, on one physical line: - [Component] <verb> <thing>.
  • A finding measured while tuning earns no bullet; prose earns one only when a docs/ file is added or removed, a restructure moves where a reader looks, or a published claim a reader acts on is corrected. docs/conventions.md -> Changelog

Scratch Files

Non-committed agent artifacts (diffs, trace outputs, generated reports, experimental scripts) go under .tmp/sessions/<session-id>/ at the repo root; .tmp/ is gitignored. <session-id> is SESSION_ID, supplied by whichever mechanism keeps it out of the cached prefix on that harness: Claude Code's SessionStart hook in .claude/settings.json prints it into the conversation ahead of the first prompt, which is already outside the system block, while OpenCode's .opencode/plugins/session-id-injector.js exports it into the environment of every shell command, because its hook would otherwise put it in the system prompt and a per-session string there destroys llama.cpp prompt-cache reuse across sessions (docs/presets.md -> Slots and the prompt cache). Under OpenCode the literal value is therefore not in context: use it inside a shell command as $env:SESSION_ID, or run Write-Output $env:SESSION_ID once when an absolute path is needed for the Write/Edit tools. If neither mechanism fired, mint YYYYMMDD-HHMMSS-<random6> instead. Never write scratch files to .claude/, the repo root, or vendor/.

Reference

Deep reference documentation lives under docs/ and is read on demand, not loaded into context. presets/README.md is the user-facing quick-start, not a reference — rationale goes under docs/. Consult the relevant file when a task touches its area:

Document When to read
docs/build_system.md Why the build scripts do what they do: submodule lifecycle, ml64.exe/vswhere toolchain detection, CUDA flags, SMT-aware parallelism, the Python requirements layering, the hardcoded upstream paths, and how speed-bench.ps1 drives a router-mode server. Read before editing rebuild_llama.cpp.ps1 or any examples/*.ps1, and before running speed-bench.ps1.
docs/presets.md Cross-model INI rules: device pinning and multi-GPU, load-mode, no-host, fit, mmproj-offload, swa-full, context shift, context size and override-kv, ngram-mod speculative decoding. Read before editing any file under presets/.
docs/conventions.md Commit message, changelog entry and pull request description, where a piece of information lives and what it may not restate, the size budgets, punctuation and table padding, .ps1 comments and PowerShell/CLI style. Read before writing a commit message, a changelog entry or a pull request description, and before adding a paragraph to AGENTS.md, docs/ or a README.
docs/model_tuning/<family>.md Per-family rationale and measured VRAM/throughput numbers, one file each: qwen.md (Qwen 3.6, 3.8, Bonsai, DSpark), qwen3.8-flash-next.md, gemma-4.md, deepseek-v4-flash.md, muse-glimmer.md, ling-3.0.md, minicpm5.md. Read the matching file before adding, retuning or removing a model entry.