Summary
When wall_time is substantially larger than user_time + system_time, the benchmarked process is spending most of its time off-CPU — blocked, waiting in the scheduler queue, or stalled in kernel paths that aren't billed to the process (memory compaction, hugepage allocation, etc). In this regime, hyperfine's mean and σ are statistically rigorous measurements of something, but that something is often not what the user thinks they're benchmarking.
Hyperfine already collects both wall and CPU times. A simple comparison would catch a class of subtle benchmarking errors that are otherwise very hard to diagnose.
Motivating example
I was benchmarking my JavaScript parser against oxc's parser on a real-world file (fabric.js, ~75k AST nodes):
Benchmark 1: ./out/joe parse fabric.js
Time (mean ± σ): 21.5 ms ± 4.2 ms [User: 3.6 ms, System: 0.7 ms]
Range (min … max): 9.9 ms … 36.9 ms 1000 runs
Benchmark 2: oxc/target/release/examples/parser fabric.js
Time (mean ± σ): 7.3 ms ± 0.7 ms [User: 3.7 ms, System: 3.3 ms]
Range (min … max): 5.8 ms … 9.6 ms 1000 runs
Summary
oxc ran 2.96 ± 0.64 times faster than ./out/joe
The obvious read: my parser is ~3× slower than oxc. Time to optimize the hot paths.
The actual situation:
- My parser's user+system time is 4.3ms, vs oxc's 7.0ms. The CPU work is less.
- The wall time (21.5ms) is 5× the CPU time. The process is off-CPU for ~17ms per run.
- Root cause: my custom allocator requests 2MB hugepages. The system's hugepage pool (
nr_hugepages=64) exhausts under hyperfine's 1000-run loop, and subsequent runs block in the kernel doing compaction/zeroing on the first page fault. This time isn't billed to the process (getrusage doesn't see it), so it appears as "missing" wall time.
After fixing the allocator's hugepage policy:
Benchmark 1: ./out/joe parse fabric.js
Time (mean ± σ): 3.8 ms ± 0.6 ms [User: 3.0 ms, System: 0.6 ms]
Range (min … max): 2.7 ms … 6.0 ms 1000 runs
Benchmark 2: oxc/target/release/examples/parser fabric.js
Time (mean ± σ): 7.4 ms ± 0.8 ms [User: 3.8 ms, System: 3.4 ms]
Range (min … max): 5.7 ms … 11.3 ms 1000 runs
Summary
./out/joe ran 1.97 ± 0.38 times faster than oxc
The actual performance comparison flipped direction. Without noticing the wall/CPU gap, I would have spent days optimizing the wrong code, and likely concluded my parser was uncompetitive.
Why this class of bug is hard to find
The signal is only visible by comparing wall time to CPU time. None of the usual debugging tools surface it on their own:
perf record / flame graphs sample on-CPU activity and are blind to off-CPU time.strace shows syscalls but missed the kernel work here (the cost was inside the page fault path, not in the mmap call itself — only 3 mmaps, ~150µs total).perf stat (single run) showed task-clock: 4.30 msec with 12 page faults — looked completely fine, because a single run got a fresh hugepage.- The bug only manifests under hyperfine's repeated-spawn pattern, which is exactly the workload hyperfine puts the user into.
The diagnostic that would have immediately pointed at the right answer is: "your process was on-CPU for 5ms but the wall clock advanced 21ms — where was it for the other 16ms?" That comparison is sitting right there in hyperfine's own output, but it's easy to miss because the eye goes to the mean/σ and not to the bracketed [User: …, System: …] line.
Proposed change
After completing a benchmark, if mean_wall > N * (mean_user + mean_system) and mean_wall is above some absolute floor (to avoid noise at the microsecond scale), emit a warning. Suggested tiers:
| Ratio |
Severity |
Suggested message |
| 2-3× |
Note |
"Some off-CPU time observed; benchmark may include scheduling effects." |
| 3-5× |
Warning |
"Substantial off-CPU time — benchmark is partly measuring system-level effects." |
| 5×+ |
Loud warning |
"Process was off-CPU for most of its runtime. This benchmark may not be measuring what you intend." |
A link to a wiki page explaining the diagnostic and common causes (CPU frequency scaling, THP/hugepage interactions, scheduler contention, process spawn overhead) would let users self-serve the investigation.
Suggested guardrails
- Absolute floor on wall time: don't warn for benchmarks under ~5-10ms unless the ratio is extreme (>10×), since user/system time has limited resolution at small scales.
- Consider per-run variance: high σ/mean combined with high wall/CPU ratio is especially diagnostic (suggests intermittent stalls rather than constant overhead).
- Comparative context: when comparing two commands, if one has a clean wall/CPU ratio and the other doesn't, that asymmetry is critical context for interpreting "X is faster than Y."
Why this fits hyperfine's design
Hyperfine already warns about measurement conditions that compromise the validity of results — for example, the "first run was significantly slower than later runs" warning that flags cold-cache effects and suggests increasing --warmup. That warning embodies the same principle this proposal extends: when hyperfine can detect from its own data that the benchmark is measuring something other than steady-state program behavior, it tells the user, because the alternative is letting them confidently draw a wrong conclusion.
The wall-vs-CPU gap fits the same mold:
- The signal comes from data hyperfine already collects — no new instrumentation.
- The phenomenon is well-understood, not a heuristic guess about user intent.
- The failure mode is silent and high-impact: users get statistically rigorous numbers that measure the wrong thing.
- More runs don't help — they just give a more confident estimate of the off-CPU stall, not of the program.
If anything, this warning is more objectively grounded than the first-run-slower one. "First run was slower" is a heuristic that sometimes catches cold caches and sometimes catches noise; "wall ≫ CPU" is a direct, mechanical observation that the process spent most of its lifetime not running.
Alternatives considered
- Doing nothing: users learn this lens eventually, but the cost of the lesson is high (wrong optimization decisions, sometimes for days).
- Documentation-only: a wiki note explaining the wall/CPU relationship. Better than nothing but won't reach people who don't think to look for it.
- Always print the ratio: simplest possible change — just add
[wall/cpu: 5.0×] next to the existing output. No threshold logic, no opinions, just makes the relationship visible. This might actually be the lowest-friction option and worth considering as an alternative to gated warnings.
Happy to discuss further or refine the proposal. Thanks for hyperfine!
Summary
When
wall_timeis substantially larger thanuser_time + system_time, the benchmarked process is spending most of its time off-CPU — blocked, waiting in the scheduler queue, or stalled in kernel paths that aren't billed to the process (memory compaction, hugepage allocation, etc). In this regime, hyperfine's mean and σ are statistically rigorous measurements of something, but that something is often not what the user thinks they're benchmarking.Hyperfine already collects both wall and CPU times. A simple comparison would catch a class of subtle benchmarking errors that are otherwise very hard to diagnose.
Motivating example
I was benchmarking my JavaScript parser against
oxc's parser on a real-world file (fabric.js, ~75k AST nodes):The obvious read: my parser is ~3× slower than oxc. Time to optimize the hot paths.
The actual situation:
nr_hugepages=64) exhausts under hyperfine's 1000-run loop, and subsequent runs block in the kernel doing compaction/zeroing on the first page fault. This time isn't billed to the process (getrusagedoesn't see it), so it appears as "missing" wall time.After fixing the allocator's hugepage policy:
The actual performance comparison flipped direction. Without noticing the wall/CPU gap, I would have spent days optimizing the wrong code, and likely concluded my parser was uncompetitive.
Why this class of bug is hard to find
The signal is only visible by comparing wall time to CPU time. None of the usual debugging tools surface it on their own:
perf record/ flame graphs sample on-CPU activity and are blind to off-CPU time.straceshows syscalls but missed the kernel work here (the cost was inside the page fault path, not in themmapcall itself — only 3 mmaps, ~150µs total).perf stat(single run) showedtask-clock: 4.30 msecwith 12 page faults — looked completely fine, because a single run got a fresh hugepage.The diagnostic that would have immediately pointed at the right answer is: "your process was on-CPU for 5ms but the wall clock advanced 21ms — where was it for the other 16ms?" That comparison is sitting right there in hyperfine's own output, but it's easy to miss because the eye goes to the mean/σ and not to the bracketed
[User: …, System: …]line.Proposed change
After completing a benchmark, if
mean_wall > N * (mean_user + mean_system)andmean_wallis above some absolute floor (to avoid noise at the microsecond scale), emit a warning. Suggested tiers:A link to a wiki page explaining the diagnostic and common causes (CPU frequency scaling, THP/hugepage interactions, scheduler contention, process spawn overhead) would let users self-serve the investigation.
Suggested guardrails
Why this fits hyperfine's design
Hyperfine already warns about measurement conditions that compromise the validity of results — for example, the "first run was significantly slower than later runs" warning that flags cold-cache effects and suggests increasing
--warmup. That warning embodies the same principle this proposal extends: when hyperfine can detect from its own data that the benchmark is measuring something other than steady-state program behavior, it tells the user, because the alternative is letting them confidently draw a wrong conclusion.The wall-vs-CPU gap fits the same mold:
If anything, this warning is more objectively grounded than the first-run-slower one. "First run was slower" is a heuristic that sometimes catches cold caches and sometimes catches noise; "wall ≫ CPU" is a direct, mechanical observation that the process spent most of its lifetime not running.
Alternatives considered
[wall/cpu: 5.0×]next to the existing output. No threshold logic, no opinions, just makes the relationship visible. This might actually be the lowest-friction option and worth considering as an alternative to gated warnings.Happy to discuss further or refine the proposal. Thanks for hyperfine!