BenchKit

Benchmark local LLMs with real evaluation suites from a full terminal UI.

Not vibes. Actual scores.

What it does

BenchKit runs established coding, reasoning, knowledge, instruction-following, and long-context benchmarks against local models. It supports OpenAI-compatible servers such as llama.cpp, llama-swap, vLLM, and LM Studio, plus native Ollama.

• Guided terminal UI for connecting, selecting models, running suites, and exploring results
• Headless mode for scripts and CI
• Live streamed responses, reasoning traces, progress, speed, and loop detection
• Direct model evaluation and optional Pi coding-agent evaluation
• Benchmark slicing, tag filters, verifier repair, and choice-order robustness
• JSON, CSV, Markdown, and standalone interactive HTML reports
• Performance profiling and a local dashboard for historical runs

Quick start

BenchKit requires Python 3.11 or newer and uv.

uv sync
cp .env.example .env
uv run benchkit

Set your server in .env, or enter it on the Connect screen:

BENCHKIT_PROVIDER=openai
BENCHKIT_HOST=http://localhost:8080/v1
BENCHKIT_API_KEY=

For Ollama:

BENCHKIT_PROVIDER=ollama
OLLAMA_HOST=http://localhost:11434

No inference server available? Explore the complete UI offline:

uv run benchkit --demo

Run benchmarks

The TUI lets you select models, benchmarks, task limits, and optional run modes. For automation, use headless mode:

# Run the first 20 HumanEval tasks
uv run benchkit --headless --models qwen3:8b --benchmarks humaneval:20

# Run multiple models and benchmarks
uv run benchkit --headless \
  --models qwen3:8b,gemma3:12b \
  --benchmarks sanity,gsm8k:50

# Print prompts, reasoning traces, and responses
uv run benchkit --headless --models qwen3:8b \
  --benchmarks sanity --verbose

Task slices accept 20 for the first 20 tasks, -20 for the last 20, and 40-80 for a range.

Useful commands:

uv run benchkit --list
uv run benchkit --help

Benchmarks

Benchmark	Key	Tasks	What it tests
Aider Polyglot	aider-polyglot	225	Repository editing across six languages with Pi
Git Surgery	git-surgery	5	Stateful Git operations with Pi
Sanity	sanity	25	Fast checks across five core capabilities
HumanEval	humaneval	164	Python function completion
HumanEval+	humaneval-plus	164	HumanEval with expanded EvalPlus tests
MBPP	mbpp	500	Short Python programming tasks
MBPP+	mbpp-plus	378	MBPP with expanded EvalPlus tests
GSM8K	gsm8k	1,319	Multi-step grade-school math
IFEval	ifeval	541	Verifiable instruction following
RULER	ruler	39 × 6	Practical long-context retrieval from 4K to 128K
RULER Full	ruler-full	6,500 × 6	Research-scale RULER (explicit opt-in)
XSTest	xstest	450	Safe compliance and unsafe refusal with an offline checker
MedFailBench	medfailbench	30	Offline heuristic screen for high-risk clinical responses; not clinical validation, certification, deployment evidence, or model ranking
GPQA	gpqa	198	Graduate-level science questions
MMLU-Pro	mmlu-pro	12,032	Reasoning across 14 knowledge categories
MMLU	mmlu	14,042	Academic and professional knowledge
ARC	arc	1,172	Grade-school science reasoning
OpenBookQA	openbookqa	500	Elementary science knowledge
WinoGrande	winogrande	1,267	Commonsense pronoun resolution
PIQA	piqa	1,838	Physical commonsense reasoning
BoolQ	boolq	3,270	Passage-based yes/no questions
TruthfulQA	truthfulqa	817	Resistance to common misconceptions
HellaSwag	hellaswag	1,000	Plausible real-world continuations

The CLI registry is the canonical source for current counts, descriptions, and supported perturbations:

uv run benchkit --list

Advanced runs

Compare raw generation with the stock Pi coding agent (Docker required):

uv run benchkit --headless --models qwen3:8b \
  --benchmarks gsm8k:20 --harness both

Give incorrect answers one verifier-guided replacement attempt:

uv run benchkit --headless --models qwen3:8b \
  --benchmarks gsm8k:20 --repair-attempts 1

Repairs default to off and can be set from 0–10. One is a practical starting point, but the right depth depends on what you want to measure.

Test whether multiple-choice accuracy survives reordered answer choices:

uv run benchkit --headless --models qwen3:8b \
  --benchmarks mmlu-pro:100 --perturbation choice-order

Profile inference performance:

uv run benchkit perf qwen3:8b
uv run benchkit perf qwen3:8b --depths minimal,4k,16k --gen 256 --reps 10

Results

Every run creates a timestamped directory containing:

results/<timestamp>/
├── results.json
├── results.csv
├── results.md
└── results.html

Open results.html for an interactive report with charts and task-level prompts, responses, reasoning traces, timing, and diagnostics.

Browse completed benchmark and performance runs locally:

uv run benchkit history

Development

uv sync
uv run pre-commit install
uv run ruff check .
uv run ruff format .
uv run pytest

See CONTRIBUTING.md for contributor setup and workflow.

License

Apache License 2.0

_{Themed with Dogi · built on Textual}