Measure local models across every runtime, and give Models its own nav section

[atomantic]

Summary

Measured Model Assessments could only benchmark Ollama and LM Studio, because its only measurement path resolved a configured PortOS provider. It now reaches every local daemon PortOS knows how to talk to — llama.cpp, MTPLX, and vLLM as well — and it records what configuration each reading describes, so "is -ub 512 worth it on this machine?" is a question you can actually answer.

Model management also stops being a scroll position on a Settings page: it gets its own top-level Models section.

Every runtime, not just the managed pair

ASSESSABLE_RUNTIMES (server/lib/localProviderRuntime.js) is the roster. The split is about how PortOS reaches the model, not about measurement quality:

Kind	Runtimes	Model list	Measurement
Managed	ollama, lmstudio	listModels() — a durable catalog on disk	runLocalLlmTest (provider path, lands in /runs)
Endpoint	llama, mtplx, vllm	GET /v1/models on the live daemon	runEndpointLlmTest — direct, no /runs record

That distinction is the whole sentinel contract for this feature. A managed backend's model list survives the daemon being down; an endpoint runtime has no catalog at all, so a stopped daemon means the list could not be read — an error, never an empty catalog. Reporting "0 models" for a daemon the user only needs to start would silently hide every model behind it.

The SSE read loop moved to server/lib/openAiChatStream.js so both paths share one implementation of the reasoning-channel handling, the skip-a-malformed-frame rule, and partial-output-on-abort.

Tuning: what PortOS can actually do with each knob

A throughput number for a GGUF is meaningless without the launch line that produced it — the same model on the same machine streams at wildly different rates depending on micro-batch size, flash attention, and KV-cache quantization. Two readings looked like noise when they were actually two different setups.

server/lib/localModelTuning.js holds the knob catalog, and every knob declares what PortOS can do with it:

• launch — PortOS puts it on the daemon's command line (llama.cpp: -b, -ub, -t, --flash-attn, --cache-type-k|v, --draft-max)
• request — sent with each measurement request (Ollama's num_ctx)
• record — PortOS cannot set it; the user states how the daemon was launched so two readings stay comparable (LM Studio, MTPLX, vLLM)

A record knob changes nothing about the run and the UI says so. What it must never do is claim to have been applied — which is why applies is on the spec rather than implied, why a request knob must also declare the wire field it maps to (both enforced by test), and why tuningApplied is null (not true) when there was nothing settable.

relaunchLlamaServerWithTuning is the launch half. It refuses rather than guesses when nothing is running (no model path to reuse) or when the process was started outside PortOS (stopping it would kill something the user owns). And because a tuning sweep is expected to produce launch lines llama.cpp rejects, a failed tuned launch restores the previous configuration before reporting — that daemon fronts the llama provider for the whole install, so leaving it down would break every later request, not just the measurement.

Store identity: no migration needed

assessmentKey(backend, modelId, tuningKey) returns backend:modelId when the tuning signature is empty and backend:modelId@<signature> otherwise. An untuned run therefore keys byte-identically to the pre-tuning key, so every record already on disk keeps resolving with no migration, while a tuned run lands beside it instead of overwriting it. Re-running the same tuning still supersedes — a stale reading of one configuration is worse than none.

compareTunings reports each variant's throughput as a percentage of the winner. Models measured under only one tuning are omitted: one reading is not a comparison, and presenting it as "the best tuning" would dress a single measurement up as a conclusion.

Models navigation

A top-level Models section at /models/:tab:

• LLMs (/models/llms) — backends, install catalog, llama.cpp launcher
• Performance (/models/performance) — measured assessments + tuning comparison
• Status (/models/status) — what is resident in memory right now
• Playground (/local-llm/playground) — unchanged path, now renders the section's tab bar so it doesn't strand you outside it

/settings/local-llm redirects to /models/llms, so bookmarks and stale ⌘K history keep working, and Local LLMs leaves the Settings sub-nav. Each tab is a route param, so all four are deep-linkable and reachable from ⌘K and voice (nav.models.performance / nav.models.status added; nav.settings.local-llm keeps its opaque id — it's in palette history — and moves to the Models section).

The llama.cpp launcher gained the same tuning flags, so a configuration can be launched and then measured. An untouched field is stripped from the payload rather than sent as 0 — llama.cpp's own default is not a value PortOS can name.

Test plan

• server/lib/localModelTuning.test.js (30 cases) — catalog invariants (every knob declares applies; every request knob declares its wire name; only llama.cpp carries launch knobs), normalize/clamp/coerce, signature stability, comparison rules
• server/services/localModelAssessments.test.js (62 cases) — endpoint-runtime listing and measurement, endpoint resolution from the live llama-server, two tunings coexisting, same-tuning replacement, per-tuning delete, tuningApplied false/null recording, runtime roster's null-vs-0 model count
• server/services/llamaServerManager.test.js — relaunch refuses when nothing is running / on an empty tuning, puts knobs on the line keeping the served model, restores the previous config when the tuned line exits, and reports not-applied when the relaunch never answers
• server/lib/openAiChatStream.test.js — SSE frame parsing, malformed-frame skip, reasoning fallback
• client/src/pages/Models.test.jsx, LocalModelAssessments.test.jsx, LocalLlmTab.test.jsx, Layout.test.jsx — tab routing from the route param, runtime roster (unreachable vs 0 models), tuning payload, warn-not-celebrate on unapplied tuning, /models full-width classification
• Full suites: server 1541 files / 32,140 tests, client 701 files / 8,866 tests, biome lint clean, client build succeeds

Follow-up

Only LLM model management lives under Models so far. LoRAs, image/video checkpoints, and embeddings are tracked in #4728. Design record: docs/plans/2026-08-21-models-navigation.md.