Feat/bench llama server and no preflight#117
Conversation
…r A/B bench/run.sh now takes --loader (vllm|llama_server) and --device (gpu|cpu) to A/B any wrapped stack against its vanilla baseline, not just vLLM-on-GPU; shared shell helpers moved into bench/lib.sh and a rawllama_entrypoint.py mirrors llama_server_infer.py's launch for the llama.cpp baseline phase. Preflight hardware auto-sizing can now be disabled via --no-preflight / MSHIP_PREFLIGHT=false (passed through to replicas alongside MSHIP_METRICS), so bench can run modelship on loader defaults + explicit config only, removing the prior "not byte-identical" caveat against the raw baseline.
The HA add-on no longer bundles the Wyoming bridge (moved to a vanilla modelship server setup), so the doc's setup instructions are stale.
Several sources of unfairness were biasing the modelship-vs-raw comparison: GPU JIT-compile toolchain gaps in the image, unwarmed/ uneven page cache between phases, llama-server inheriting every GPU on a multi-GPU host instead of just its reservation, connection resets under concurrency on the llama_server CPU baseline, and GGUF configs having no HF tokenizer id for the bench client. Add launch-parity and result-parity gates that fail the run before summarizing if the two arms didn't actually run identical engine args or dropped requests.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a --no-preflight option (MSHIP_PREFLIGHT=false) to disable hardware auto-sizing during benchmarking, updates the Dockerfile with dependencies for JIT-compiling kernels on newer GPU architectures, and refactors the benchmarking harness to support both vllm and llama_server loaders across CPU and GPU devices. Review feedback recommends expanding the model cache pre-warming helper to include .safetensors and other formats to ensure fair A/B comparisons on the vllm loader, and passing the /v1/models JSON response via standard input to avoid command-line argument length limits.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
The llama_server A/B occasionally saw modelship "truncate" a request: with --ignore-eos the model can babble a malformed <tool_call> past EOS that llama-server's own grammar parser rejects mid-stream, which modelship faithfully relays as an in-band SSE error. This is engine behaviour present in both arms; under sampling it landed on one arm by luck. - Pin --temperature 0 in the vllm-bench load client so both arms decode an identical deterministic token stream — the A/B is reproducible and any shared engine-level in-band error appears symmetrically instead of randomly. - Rewrite assert_result_parity to compare drop/truncation counts RELATIVE between arms: modelship dropping more hard-fails; the baseline dropping more is a FINDING (run passes); equal is parity. The old absolute per-arm gate could mis-attribute an upstream engine artifact to modelship's wrapping. - Document a llama_server / GPU results table in bench/README.md mirroring the vllm one, and drop the peak-host-RAM row from both tables. - Update the config NOTE comments to match the relative gate.
1 participant
No description provided.