Performance profile: measured acceleration headroom + decomposition (#56)#60
Merged
Conversation
…mposition (#56) Profile-before-optimize, per #56's own mandate — measured-never-asserted applied to performance: - evals/look-up/profile_mcp.py: speaks MCP stdio (newline-delimited JSON-RPC) directly to the installed signed binary — engine-side timings, no agent loop. Cold→warm within one server lifetime + fresh-process cold. - evals/look-up/perf_profile.md: dated capture (2026-07-06) with reproduce commands + loud caveats (small n, one machine/network — directional). Key numbers: warm ETag 10× (344→32 ms) but per-process only; sequential probing (zero TaskGroup/async-let in Sources/) makes fastapi resolve ~3× react (1055–1159 vs 353–385 ms); haiku headless spawn ≈9 s — refutes subagent delegation for single-shot lookups (8–25× net loss); look-up docs questions cost 2 main-loop turns, which dominate end-to-end latency. - README pointer. Children filed from the measurements: #57 (TaskGroup parallel probing), #58 (disk-persisted bounded ETag cache — local-only, inside the anti-positioning boundary), #59 (composite one-call lookup, turns 2→1 — the measured replacement for the refuted single-shot delegation). Python suite untouched (72 green).
…s per #56 verify (#56) Two pinned-opus lenses (measurement-honesty + scope/regression) reviewed the profile; their findings materially improved the measurement: - MEDIUM: the original "cold" resolve_source react ran AFTER latest_version calls had ETag-/connection-warmed the registry leg → understated cold by ~2x (367 vs the unbiased 649 ms). Harness reordered (each library's resolve runs first in its process); table replaced with the corrected capture; the biased first capture is kept in a "capture note" as a worked example of why single runs mislead. - Consequence: fastapi is ~1.8x react (1146 vs 649), NOT 3x — causal claim softened to "consistent with summed probes; harness cannot decompose per-probe vs host latency; #57 must instrument per-probe timing before projecting gains". - MEDIUM: "largest MEASURED lever" mislabeled the composite-call lever — its magnitude is the per-turn cost this engine-side harness deliberately does not measure. Relabeled "expected"; section ② now splits measured (tool time) from asserted (turn cost) explicitly. - LOW: warm-win attribution conceded to bundle connection/DNS/TLS reuse with ETag (observed 2-10x across runs); #58 implication stated (disk cache recovers only the revalidation share). - LOW: 8-25x arithmetic corrected to ~7-30x (9/1.2=7.5, 9/0.3=30); claude -p proxy for the Agent tool explicitly labeled an assertion. - LOW (scope lens): graceful missing-binary error (friendly hint, exit 1) instead of a raw FileNotFoundError traceback. - New "stable vs noise" section separates structural facts (source-derived) from timings (network-dominated, wide same-day spreads). Children #57/#58/#59 bodies synced to the corrected numbers. 72 Python green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #56
Summary
#56 asked: within the local+direct architecture (no central cache), where's the acceleration headroom — and can subagents/haiku help? This PR delivers what the issue's Expected section demands: a dated, reproducible profile + a measured decomposition into children.
Measured (2026-07-06, directional — reproduce:
python3 evals/look-up/profile_mcp.py)TaskGroup/async lethits) → sequential probe chains;resolve_source fastapi1055–1159 ms vs react 353–385 ms. Warm ETag 10× (344→32 ms) but per-process only (docstring admits it).Changes
evals/look-up/profile_mcp.py— MCP-stdio timing harness (engine-side, no agent loop)evals/look-up/perf_profile.md— dated capture + reproduce commands + loud caveatsevals/look-up/README.md— pointerNo engine code changes here (those are the children). Python suite untouched: 72 green; committed harness re-run confirms the capture's shape.
Checklist
Refs #56)🤖 /idd-all · Do NOT add Closes/Fixes/Resolves.